KIT | KIT-Bibliothek | Impressum | Datenschutz

3D-Guided Face Manipulation of 2D Images for the Prediction of Post-Operative Outcome after Cranio-Maxillofacial Surgery

Andlauer, Robin 1; Wachter, Andreas ORCID iD icon 1; Schaufelberger, Matthias 1; Weichel, F.; Kühle, R.; Freudlsperger, Christian; Nahm, Werner 1
1 Institut für Biomedizinische Technik (IBT), Karlsruher Institut für Technologie (KIT)


Cranio-maxillofacial surgery often alters the aesthetics of the face which can be a heavy burden for patients to decide whether or not to undergo surgery. Today, physicians can predict the post-operative face using surgery planning tools to support the patient's decision-making. While these planning tools allow a simulation of the post-operative face, the facial texture must usually be captured by another 3D texture scan and subsequently mapped on the simulated face. This approach often results in face predictions that do not appear realistic or lively looking and are therefore ill-suited to guide the patient's decision-making. Instead, we propose a method using a generative adversarial network to modify a facial image according to a 3D soft-tissue estimation of the post-operative face. To circumvent the lack of available data pairs between pre- and post-operative measurements we propose a semi-supervised training strategy using cycle losses that only requires paired open-source data of images and 3D surfaces of the face's shape. After training on "in-the-wild" images we show that our model can realistically manipulate local regions of a face in a 2D image based on a modified 3D shape. ... mehr

Verlagsausgabe §
DOI: 10.5445/IR/1000137476
Veröffentlicht am 01.10.2021
DOI: 10.1109/TIP.2021.3096081
Zitationen: 1
Web of Science
Zitationen: 1
Zitationen: 3
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Biomedizinische Technik (IBT)
Publikationstyp Zeitschriftenaufsatz
Publikationsjahr 2021
Sprache Englisch
Identifikator ISSN: 1057-7149, 1941-0042
KITopen-ID: 1000137476
Erschienen in IEEE Transactions on Image Processing
Verlag Institute of Electrical and Electronics Engineers (IEEE)
Band 30
Seiten 7349-7363
Vorab online veröffentlicht am 15.07.2021
Nachgewiesen in Scopus
Web of Science
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page