Quest for quality: a review of design knowledge on gamified AI training data annotation systems

Data annotation is a tedious, yet vital task to create AI training data. Gamification can support annotation quality by motivating and engaging annotators. Existing gamified annotation system artifacts are largely instantiations; it remains unclear how the ingrained design knowledge transcends individual and situational factors and can inform the design of gamified annotation systems as a system class. This study synthesises extant gamified annotation system artifacts and gauges the maturity of ingrained design knowledge. We conduct a semi-systematic review of 56 articles which present gamified annotation system artifacts. Beyond a broad overview of design artifacts and research activities, we derive 13 solution streams that describe design knowledge as means-end relationships between goals and gamification-based solutions. While some solution streams exhibit high maturity, the results largely confirm our initial assumption that the design knowledge base on gamified annotation systems is immature. To advance maturity, we recommend to move beyond creating new instantiations as the primary research activity and to focus on developing actionable design prescriptions. ... mehrOur study contributes to creating more mature design knowledge on gamified annotation systems and offers valuable perspectives on the maturity and scholarly progression of gamification research.