We are very excited to join forces with MLCommons and OctoML.ai! Contact Grigori Fursin for more details!

A Realistic Face-to-Face Conversation System based on Deep Neural Networks

lib:e09ac1d886801a84 (v1.0.0)

Authors: Zezhou Chen,Zhaoxiang Liu,Huan Hu,Jinqiang Bai,Shiguo Lian,Fuyuan Shi,Kai Wang
ArXiv: 1908.07750
Document:  PDF  DOI 
Abstract URL: https://arxiv.org/abs/1908.07750v1


To improve the experiences of face-to-face conversation with avatar, this paper presents a novel conversation system. It is composed of two sequence-to-sequence models respectively for listening and speaking and a Generative Adversarial Network (GAN) based realistic avatar synthesizer. The models exploit the facial action and head pose to learn natural human reactions. Based on the models' output, the synthesizer uses the Pixel2Pixel model to generate realistic facial images. To show the improvement of our system, we use a 3D model based avatar driving scheme as a reference. We train and evaluate our neural networks with the data from ESPN shows. Experimental results show that our conversation system can generate natural facial reactions and realistic facial images.

Relevant initiatives  

Related knowledge about this paper Reproduced results (crowd-benchmarking and competitions) Artifact and reproducibility checklists Common formats for research projects and shared artifacts Reproducibility initiatives

Comments  

Please log in to add your comments!
If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!