Short abstract:

A critical genealogy and social theory of data annotation: we propose a framework for understanding the interplay of the global social conditions of data annotation with the subjective phenomenological experience of data annotation work.

Long abstract:

Data annotation remains the sine qua non of machine learning and AI. Recent work on data annotation highlights the importance of rater diversity for fairness, model performance, and new lines of research have begun to examine the working conditions for data annotation workers, the impacts and role of annotator subjectivity on labels Data annotation has become a global industry. This paper outlines a critical genealogy of data annotation; starting with its psychological and perceptual aspects. We draw on similarities with critiques of the rise of computerized lab-based psychological experiments in the 1970’s which question whether these experiments permit the generalization of results beyond the laboratory settings within which these results are typically obtained. Similarly, do data annotations permit the generalization of results beyond the settings, or locations, in which they were obtained? Moreover, Western psychology is overly reliant on participants from Western, Educated, Industrialized, Rich, and Democratic societies (WEIRD). Many of the people who work as data annotation platform workers, however, are not from WEIRD countries; most data annotation workers are based in Global South countries. Social categorizations and classifications from WEIRD countries are imposed on non-WEIRD annotators through instructions and tasks, and through them, on data, which is then used to train or evaluate AI models in WEIRD countries. What does it mean for non-WEIRD workers to annotate data from and about WEIRD societies? We propose a framework for understanding the interplay of the global social conditions of data annotation with the subjective phenomenological experience of data annotation

work.