Paintings, not Noise - The Role of Variety and Self-Determination in Labeling Work and its Effects on Psychological and Performance Outcomes

Labeling is critical in creating training datasets for supervised machine learning. It typically requires manual labor, is costly, and not infrequently frustrates the workers involved. Although task variety drives human autonomy and intrinsic motivation, there is little research in this regard in the labeling context. Against this backdrop, we manipulate the presentation sequence of a labeling task in an online experiment using the theoretical lens of self-determination theory to examine workers’ perceptions of task variety, autonomy, and intrinsic motivation, as well as the effects on psychological and performance work outcomes. We rely on 180 crowd workers contributing with group comparisons between three presentation sequences (by image, by label, random) and a mediation path analysis along the phenomena studied. Our results show, how task variety as perceived by workers does not equal mathematical task variety as in randomization. Rather, labeling workers seem to be better accompanied in more structured approaches to the sequence of presentation. We choose a visual metaphor to explain this phenomenon, whereas paintings are but a structured re-arrangement of coloured pixels, as opposed to random noise.