KIT | KIT-Bibliothek | Impressum | Datenschutz

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Waibel, Alexander 1; Behr, Moritz 1; Yaman, Dogucan 1; Eyiokur, Fevziye Irem 1; Nguyen, Tuan-Nam 1; Mullov, Carlos 1; Demirtas, Mehmet Arif; Kantarci, Alperen; Constantin, Stefan 1; Ekenel, Hazim Kemal
1 Institut für Anthropomatik und Robotik (IAR), Karlsruher Institut für Technologie (KIT)

Abstract:

In this paper, we propose a neural end-to-end system for voice preserving, lip-synchronous translation of videos. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, face video of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by a translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases mapped from the original sentence. The resulting synthetic voice is then mapped back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a conditional generative adversarial network-based model generates frames of adapted lip movements with respect to the input face image as well as the output of the voice conversion model. In the end, the system combines the generated video with the converted audio to produce the final output. The result is a video of a speaker speaking in another language without actually knowing it. ... mehr


Volltext §
DOI: 10.5445/IR/1000161934
Veröffentlicht am 05.09.2023
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Anthropomatik und Robotik (IAR)
Publikationstyp Forschungsbericht/Preprint
Publikationsjahr 2022
Sprache Englisch
Identifikator KITopen-ID: 1000161934
Umfang 14 S.
Vorab online veröffentlicht am 09.06.2022
Nachgewiesen in arXiv
Dimensions
Relationen in KITopen
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page