Lyrics Display for Live Chorus Performances.
MOTIVATION
Live musical performances (e.g., choruses, concerts, and operas) often require the display of lyrics for the convenience of the audience. This requires following the performance and controlling the lyrics display in real time. Traditionally, this is usually controlled by a staff member of the concert who has been very familiar with the performance. This process requires great labor intensity, since the staff member needs to pay attention and scroll down the lyrics at the correct timing through the whole concert. Alternatively, we aim to propose a computational system to automate this real-time lyrics display process using signal processing techniques.
THE SYSTEM
The system records the live audio of the concert performance, analyzes it to follow the progress, and displays each lyric sentence to public at the correct timings. More specifically, it consists of 3 modules:
Real-time Audio Recording
The system keeps recording the whole concert on a separate thread and keeps passing the recorded audio samples (as waveform) to the algorithm on each 46ms audio buffer. This results in a minimum delay of 46ms for scrolling down the lyrics, but is hardly noticeable.
Algorithm
This is the core function of the system, which parses the recorded audio waveforms, outputs the estimated progress of the current song, and thus displays the corresponding lyric sentence on the interface. This analysis is implemented by comparing the recorded live audio stream with a reference recording of the same piece. The reference audio can be a public version downloaded from YouTube, or collected from the same chorus during a rehearsal. The index timing of each lyric sentence should be pre-encoded on the reference audio. During the live concert, the algorithm only analyzes the pitch/chord characteristics of the signal instead of directly recognizing the lyric words from the recorded phonemes, which guarantees a more robust framework.
Graphic User Interface
The software implementing the system provides two GUIs. One is for the audience that only displays one single lyric sentence of the current song and is compatible with a projector display. The other only shows on the operating system that provides interface of several basic functions, such as load reference audio, start, stop. It also provides optional manual control of the system, such as pause or forward/reverse 3 sec, to prevent unexpected errors of the algorithm running. The whole system is developed by Java and is able to install on personal laptops.
A demonstration of the system can be referred here:
HHOW TO USE
Before the Concert
After confirming the performance piece list, we need to collect the reference audio and the corresponding encoded lyrics timings manually. Similar to a Karaoke system, we intended to align the lyrics index a little earlier (e.g., 1 sec) than the actual occurrence timings on the reference audio. This results in an earlier fashion of the lyrics scrolling during the live concert, for a better enjoyment experience to the audience. Note that although this annotation process is a manual work, this only needs to be done once before the live concert and is good to use for all the performances in the future. We also provide a web service to simplify this annotation process and store a number of pre-encoded pieces HERE.
During the Concert
We let the system run on a personal laptop and connect it with a projector. Using the software interface, we load all the reference audio (with the encoded lyrics) following the performance order. Upon each song’s start, the staff member click the “start” function to enable the recording and algorithm running. It will stop automatically at the end of each song.
USAGE EXPERIENCE
This project starts from 2015 and the system has been developed with the collaboration of the CCSR. The initial version of the system was tested on April 30, 2016 during the annual concert of CCSR. In the following year, the system was debut to public and completely replaced the labor work for the annual concert on May 13, 2017. On April 28, 2018, we presented the system to public again with a more robust framework.
During the concerts, we used a MacBook Pro to run the software. The device locates in the center of the auditorium along with other controllers of the concert, such as lights, speakers, projector, etc. The live recording was collected using the built-in microphone of the MacBook instead of the stand-alone microphones on the stage. Although the interface allows some manual adjustment of the lyrics display as auxiliary functions, these were rarely used during all the previous tests, reported by the developers. This means the algorithm is robust to noise and reverberations in real concert scenarios.
Here is a sample demo video recorded from the live concert on April 28, 2018.
The full playlist of the concert on April 28, 2018 can be accessed HERE.