DECM: Evaluating Bilingual ASR Performance on a Code-switching/mixing Benchmark

Automatic Speech Recognition has made significant progress, but challenges persist. Code-switched (CSW) Speech
presents one such challenge, involving the mixing of multiple languages by a speaker. Even when multilingual ASR
models are trained, each utterance on its own usually remains monolingual. We introduce an evaluation dataset for
German-English CSW, with German as the matrix language and English as the embedded language. The dataset
comprises spontaneous speech from diverse domains, enabling realistic CSW evaluation in German-English. It
includes splits with varying degrees of CSW to facilitate specialized model analysis. As it is difficult to collect CSW
data for all language pairs, the provision of such evaluation data, is crucial for developing and analyzing ASR models
capable of generalizing across unseen pairs. Detailed data statistics are presented, and state-of-the-art (SOTA)
multilingual models are evaluated showing challanges of CSW speech.