KIT | KIT-Bibliothek | Impressum | Datenschutz

F-Actor: Controllable Conversational Behaviour in Full-Duplex Models

Züfle, Maike ORCID iD icon 1; Klejch, Ondrej; Sanders, Nicholas; Niehues, Jan ORCID iD icon; Birch, Alexandra; Lam, Tsz Kin
1 Institut für Anthropomatik und Robotik (IAR), Karlsruher Institut für Technologie (KIT)

Abstract:

Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviour that adapts dynamically to the context. Current spoken conversational systems, however, rarely allow such customization, limiting their naturalness and usability. In this work, we present the first open, instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints. By keeping the audio encoder frozen and finetuning only the language model, our model requires just 2,000 hours of data, without relying on large-scale pretraining or multistage optimization. The model can follow explicit instructions to control speaker voice, conversation topic, conversational behaviour (e.g., backchanneling and interruptions), and dialogue initiation. We propose a single-stage training protocol and systematically analyze design choices. Both the model and training code is released to enable reproducible research on controllable full-duplex speech systems.


Zugehörige Institution(en) am KIT Institut für Anthropomatik und Robotik (IAR)
Publikationstyp Proceedingsbeitrag
Publikationsjahr 2026
Sprache Englisch
Identifikator ISBN: 979-8-89176-395-1
ISSN: 0736-587X
KITopen-ID: 1000198222
Erschienen in Proceedings of the Annual Meeting of the Association for Computational Linguistics
Veranstaltung 64th Annual Meeting of the Association for Computational Linguistics (2026), San Diego, CA, USA, 02.07.2026 – 07.07.2026
Verlag Association for Computational Linguistics (ACL)
Seiten 4904 - 4921
Externe Relationen Siehe auch
Nachgewiesen in Scopus
KIT – Die Universität in der Helmholtz-Gemeinschaft
KITopen Landing Page