Song lyrics in real time, multiple languages
New system displays song lyrics in real time, multiple languages
A recent choral concert reflecting on the past with the theme “Memories,” also featured the debut of a new technological advancement in music.
The Chinese Choral Society of Rochester presented a concert that included Chinese and English art and folk songs and also marked the launch of an automatic lyrics display system developed by Zhiyao Duan, an assistant professor of electrical and computer engineering at the University of Rochester, and his PhD student, Bochen Li. The system is able to follow a performance and display lyrics on a large screen in real time.
During the May concert, the system displayed the lyrics in both Chinese and English to a multicultural audience composed of people with a variety of native languages.
The Chinese Choral Society of Rochester performs, with lyrics displayed in real-time in both English and Chinese. (University photo / Yichi Zhang )
“Musical performance is all about communication between musicians and the audience,” Duan says. “This system helps musicians convey ideas—in this case the lyrics—to the audience and helps the audience better appreciate the performance.”
Duan sings tenor with the Chinese Choral Society and brainstormed with Youti Kuo, secretary of the choral society, and Ming-Lun Lee, an assistant professor of audio and music engineering at the University of Rochester and the director of the choral society, about ways he could also contribute his knowledge as a researcher in artificial intelligence and music.
Rather than relying on someone to manually type song lyrics during a live performance, professor Zhiyao Duan’s system uses an algorithm to compare a reference recording with a live performance and display the matching song lyrics automatically. (University photo / Yichi Zhang)
The choir performs in various languages, and previously, in order to display lyrics in real time, a person very familiar with the songs would have to manually type the words during the live performance. Duan’s innovation allows an audience to view the lyrics at the exact moment they are performed, without the need for manual lyric input during the performance.
“It is very important to have the audience understand the lyrics in foreign languages,” says Lee. “With the help of the automatic lyrics displaying system, we can train the system before the concert and avoid human errors.”
Duan’s system uses a dynamic time warping algorithm, which reconciles differences between two sequences that may vary in speed. For the system to work, it needs a reference recording to compare with the live performance. The choral society taped a reference recording during a rehearsal and pre-aligned the lyrics to this recording. During the live performance, the algorithm aligned the live performance with the reference recording and displayed the lyrics in real time.
Although manual work is still required to add lyrics to the reference recording prior to the concert, it can be completed in a much more flexible fashion: it can be done any time before the concert and allows for revisions. In addition, once the lyrics alignment is completed, it can be used as a reference for any live performance of the same piece.
“We hope this will soon be available to choirs worldwide, either for commercial use or for free,” Duan says. “We are working on building a system and a repertoire that are scalable.”
In addition to the debut of the lyrics system, the concert featured the US premiere of the suite A Journey Named Life for chorus, flute, and piano, composed by Ching-Mei Lin ’06 (MM).
—Lindsey Valich, May 2017
