Paper short abstract:

AI systems require vast amounts of labour to develop and maintain, with data annotation playing a key role in these. This empirical study investigates practitioner perspectives and expectations regarding data annotation, promoting critical reflection around wider machine learning practices.

Paper long abstract:

A vast amount of human labour is required to develop and maintain AI models and systems, with data annotation playing a central role. However, this is often overlooked in the discourse around technological innovation and responsible AI. Furthermore, such work is conducted at the intersection of multiple professional groups who often have little visibility, such as gig economy workers. This labour is not just invisible to users of AI, its mechanisms can be partially obscured from the view of machine learning practitioners who use those data annotation services. Our research addresses this gap, mapping the points of contact which practitioners have with annotators, and their perceptions of annotators and annotation companies. Building on literature around crowdsourced data work, we challenge dominant narratives on automation by centering the invisible labour behind the functioning of many AI technologies and exploring methods for the creation of more participatory and democratic machine learning systems. Our empirical work investigates machine learning practitioner perspectives and expectations regarding data annotation work. Drawing from workshops conducted with machine learning practitioners, we explore the collaborative practices of data work, particularly the points of contact between data workers with different levels and types of expertise. We focus on experiences of data ‘wrangling’, or practices of data acquisition, labelling, and cleaning, as the point where researchers and engineers interface with domain experts, annotators, and other workers. In addition to contributing to understanding of the hidden labour involved in data wrangling, we aim to promote critical reflection around wider machine learning practices.