Increasing Label Quality with Interactive Image Labeling Systems: An Explanatory Design Theory

Despite rapid progress in the field of deep learning, image labeling remains a core requirement for any organization looking to use machine learning for image recognition and generation to solve real-world business problems. Domain-specific training data often needs to be provided by human data contributors who are tasked with manually providing labels. Unfortunately, there is limited research on the design of image labeling systems that emphasize the perspective of data contributors, who are essential to achieving high-quality training data. Drawing on the theory of interactive media effects, we performed a comprehensive design science research project following the design explanatory design theory genre: We derived four design requirements for interactive image labeling systems that support interactivity-based improvements in label quality. We evaluated our proposed design in two evaluation episodes and demonstrated its positive impact on label quality in a real-world context with crowdworkers. In addition to these design requirements, we propose two theoretically grounded and empirically evaluated design principles for interactive image labeling systems, which address the issue of poorly designed data collection systems that do not support data contributors in delivering high-quality data. ... mehrNotably, the study contributes to existing research by providing a solution to the critical challenge of collecting high-quality labels for artificial intelligence-based information systems. Furthermore, we illustrate how the genre of explanatory design theory can bridge the gap between design science research and empirical research. On this basis, our research offers a blueprint for embedding empirical findings into explanatory design theory to generate prescriptive knowledge.