Gamified Expert Annotation Systems: Meta-Requirements and Tentative Design

Poorly annotated data is a common problem for data-intensive applications like supervised machine learning. In domains like healthcare, annotation tasks require specific domain knowledge and are thus often done manually
by experts, which is error-prone, time-intensive, and tedious. In this study, we
investigate gamification as a means to foster annotation quality through annotators’ increased motivation and engagement. To this end, we conducted a literature
review of 70 studies as well as a series of 16 workshops with a team of six experts in
medical image annotation. We derive a set of seven meta-requirements (MRs) that
represent the desired instrumental and experiential outcomes of gamified expert
annotation systems (e.g., high-quality annotations, a sense of challenge) as well
as a tentative design that can address the derived MRs. Our results help to understand the inner workings of gamification in the context of expert annotation and
lay important groundwork for designing gamified expert annotation systems that
can successfully motivate annotators and increase annotation quality.