Song lyrics in real time, multiple languages

Song lyrics in real time, multiple languages

New system displays song lyrics in real time, multiple languages

A recent choral concert reflecting on the past with the theme “Memories,” also featured the debut of a new technological advancement in music.

The Chinese Choral Society of Rochester presented a concert that included Chinese and English art and folk songs and also marked the launch of an automatic lyrics display system developed by Zhiyao Duan, an assistant professor of electrical and computer engineering at the University of Rochester, and his PhD student, Bochen Li. The system is able to follow a performance and display lyrics on a large screen in real time.

During the May concert, the system displayed the lyrics in both Chinese and English to a multicultural audience composed of people with a variety of native languages.

The Chinese Choral Society of Rochester performing

The Chinese Choral Society of Rochester performs, with lyrics displayed in real-time in both English and Chinese. (University photo / Yichi Zhang )

“Musical performance is all about communication between musicians and the audience,” Duan says. “This system helps musicians convey ideas—in this case the lyrics—to the audience and helps the audience better appreciate the performance.”

Duan sings tenor with the Chinese Choral Society and brainstormed with Youti Kuo, secretary of the choral society, and Ming-Lun Lee, an assistant professor of audio and music engineering at the University of Rochester and the director of the choral society, about ways he could also contribute his knowledge as a researcher in artificial intelligence and music.

image of laptop with Zhiyao Duan's system

Rather than relying on someone to manually type song lyrics during a live performance, professor Zhiyao Duan’s system uses an algorithm to compare a reference recording with a live performance and display the matching song lyrics automatically. (University photo / Yichi Zhang)

The choir performs in various languages, and previously, in order to display lyrics in real time, a person very familiar with the songs would have to manually type the words during the live performance. Duan’s innovation allows an audience to view the lyrics at the exact moment they are performed, without the need for manual lyric input during the performance.

“It is very important to have the audience understand the lyrics in foreign languages,” says Lee. “With the help of the automatic lyrics displaying system, we can train the system before the concert and avoid human errors.”

Duan’s system uses a dynamic time warping algorithm, which reconciles differences between two sequences that may vary in speed. For the system to work, it needs a reference recording to compare with the live performance. The choral society taped a reference recording during a rehearsal and pre-aligned the lyrics to this recording. During the live performance, the algorithm aligned the live performance with the reference recording and displayed the lyrics in real time.

Although manual work is still required to add lyrics to the reference recording prior to the concert, it can be completed in a much more flexible fashion: it can be done any time before the concert and allows for revisions. In addition, once the lyrics alignment is completed, it can be used as a reference for any live performance of the same piece.

“We hope this will soon be available to choirs worldwide, either for commercial use or for free,” Duan says. “We are working on building a system and a repertoire that are scalable.”

In addition to the debut of the lyrics system, the concert featured the US premiere of the suite A Journey Named Life for chorus, flute, and piano, composed by Ching-Mei Lin ’06 (MM).

—Lindsey Valich, May 2017


More Stories

  • We met here

    Many University of Rochester love stories began in the halls, tunnels, and green spaces around our campuses. Explore this interactive map to learn more about some of these couples and how and where they met....

  • an asian man is standing in the middle of a hall as he is leading an orchestra of students

    Powerful performances for artistry ever better

    When industrialist George Eastman purchased the property on Gibbs and Main Street to build a school of music and theatre, one of his primary goals was to provide the citizens of Rochester with a place to study, make, and enjoy music. Etched...

  • A female nurse checking a patients heartbeat

    Breaking down barriers to healthcare for outcomes ever better

    Nurse practitioners at the School of Nursing provide free health services to students in the City of Rochester....

  • Woman in white cap and gown holding her diploma over her head on stage during graduation

    Transforming education for futures ever better.

    Rochester’s Warner School of Education partnered with East Upper and Lower Schools to transform the classroom and the community....