Speech Driven Talking Face Generation from a Single Image and an Emotion Condition

[paper link][code link]

This project is supported by the National Science Foundation under grant No. 1741472, titled "BIGDATA: F: Audio-Visual Scene Understanding". Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

What is the problem?

Speech communication does not solely depend on the acoustic signal. Visual cues, when present, also play a vital role. The presence of visual cues improves speech comprehension in noisy environments and for the hard-of-hearing population. Consequently, researchers developed systems that can automatically generate talking faces from speech in order to provide the visual cues when they are not available. These systems can increase the accessibility of abundantly available audio-only resources for the hearing impaired population. They can also find wide applications in entertainment, education, and healthcare.

During speech communication, emotion has a direct impact on the transmitted message and can change the meaning drastically [1]. Studies have shown that predicting emotions purely from speech audio is quite difficult for untrained people [2] and that we heavily rely on visual cues in emotion interpretation [3]. Therefore, to make the visual rendering more realistic and to improve speech communication, it is important for automatic talking face generation systems to render visual emotion expressions.

What is our approach?

Instead of inferring emotion from the input speech, in this work, we propose to use emotions as a condition input to our system. The motivation is to decouple the speech and emotion conditions. This allows us to manipulate emotions during the generation of face videos. The figure below shows the system overview, which employs the generative adversarial network (GAN) framework. Our generator network architecture is built based on our previous work [4], with a modification to accept the emotion condition input. For discriminator networks, we use one discriminator to distinguish the emotions expressed in videos, and another discriminator to distinguish the real and generated video frames.

Our Results

The following examples are generated from CREMA-D test samples. The image and speech input to the network are unseen from training.

Generated videos conditioned on the speech emotion label (i.e., generated visual emotion matches that of speech)