Send message to Authors

Paper short abstract

This paper presents a research protocol to study how workers use LLMs in practice. Through a six-month ethnographic studio inquiry with 32 participants, we developed a method for observing LLMs-in-use and identifying their consequences for work, challenging deterministic accounts of AI’s impact.

Paper long abstract

Three years after the introduction of ChatGPT, around 40% of people in the US and Europe use Large Language Models (LLMs), yet we still know little about what it means to “use” LLMs at work. Much of the current discussion focuses on predicting the impact of AI or extrapolates from the properties of LLMs (e.g., bias or confabulation) instead of studying how they are enacted in ordinary practices. Our paper presents the protocol we implemented to examine what users value in these systems and what they construe as problems.

We revisit the ethnographic tradition of studying technologies-in-use (Suchman 1999; Orlikowski 2000) to address the methodological challenges of GenAI. Observing LLM use is difficult because interactions are individualized, locked behind proprietary interfaces, and embedded in a normatively saturated environment that encourages workers to report what they should do rather than what they actually do. Describing these systems also poses difficulties for STS scholars, as their probabilistic and task-agnostic nature challenges classical notions such as the “script” (Akrich 1992).

To address these challenges, we designed a research “studio” in which workers participated as co-inquirers. Using a structured exercise protocol spanning six months, participants brought conversation logs, described their practices, and collectively reflected on concrete situations. Four cohorts of workers (32 in total) followed this protocol. This protocol allowed us to expand our analytical repertoire and to identify several consequences of LLMs-in-use. It also helps deflate deterministic narratives about AI’s impact on work while decentering technologists’ perspectives by foregrounding workers’ experiences.