MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations

Sarcasm is a complex form of figurative language in which the intended meaning contradicts the literal one. Its
prevalence in social media and popular culture poses persistent challenges for natural language understanding,
sentiment analysis, and content moderation. With the emergence of multimodal large language models, sarcasm
detection extends beyond text and requires integrating cues from audio and vision. We present MuSaG, the first
German multimodal sarcasm detection dataset, consisting of 33 minutes of manually selected and human-annotated
statements from German television shows. Each instance provides aligned text, audio, and video modalities,
annotated separately by humans, enabling evaluation in unimodal and multimodal settings. We benchmark nine
open-source and commercial models, spanning text, audio, vision, and multimodal architectures, and compare their
performance to human annotations. Our results show that while humans rely heavily on audio in conversational
settings, models perform best on text. This highlights a gap in current multimodal models and motivates the use of
MuSaG for developing models better suited to realistic scenarios. ... mehrWe release MuSaG publicly to support future
research on multimodal sarcasm detection and human–model alignment.