Virtual Listener Using Conditional Motion Synthesis     

In this project, a photorealistic virtual character will learn how to listen attentively to a speaker in real-time and respond appropriately to the speaker. This can include, for example, nonverbal behavior such as backchanneling (e.g., nodding, smiling), but also the appropriate facial expressions or gestures for the current context. Meta-Human is used for visualization (Epic Games), and state-of-the-art methods from the literature are used for machine learning. The end result is a live pipeline in which the user tells a story and the virtual character applies the learned listener behavior in the meantime.

Virtual Speaker Using Conditional Motion Synthesis     

In this project, a photorealistic virtual character should learn how to show appropriate nonverbal behavior when speaking. Since this behavior sometimes depends on the context (e.g. emotional vs non-emotional), the facial expressions or gestures for the current speech sequence may vary.  Meta-Human is used for visualization (Epic Games), and state-of-the-art methods from the literature are used for machine learning. The final result is a live pipeline in which the virtual character delivers a story together with the learned nonverbal behavior.

Discuss with a Virtual Character     

In this project, a pipeline should be created in which a user can have a conversation with a photorealistic character. The character should give appropriate answers (e.g. LLM) and speak with synchronous lip movements. Most of the tools for this already exist at the chair and just need to be assembled. A main focus will be to use an already existing TTS system to generate lip movements and audio.