Send message to Authors

Paper short abstract

This paper explores methodological challenges in studying LLMs through an operational ethnography of GPT. By experimenting with prompts during the sentiment analysis of YouTube comments, the study reveals methodological biases, opacity, and interpretive mediation in AI assisted research.

Paper long abstract

As Large Language Models (LLMs) increasingly shape research practices in the social sciences, they are often approached as efficient tools capable of automating tasks such as text generation, classification, or data analysis. However, this instrumental perspective risks overlooking the complex ways in which these systems participate in and reshape methodological practices.

This paper addresses the methodological challenges of studying contemporary AI systems by proposing an operational ethnography of GPT, both through its chat interface and its API. The study examines the model’s behaviour in a real research setting: the sentiment analysis and visualization of YouTube comments. Rather than treating the model as a neutral analytical instrument, the research focuses on the interaction between researcher, prompts, data, and model outputs.

Through systematic prompt experimentation, the analysis shows that working with LLMs involves continuous iterations, adjustments, and interpretive decisions. Small variations in prompt design generate different classification outcomes, revealing how LLM based analyses are shaped by methodological biases, system opacity, and infrastructural constraints.

Building on debates in Science and Technology Studies and critical AI studies, the paper argues that LLMs should be understood as hybrid methodological entities that simultaneously function as tools, devices, agents, and opaque “pseudo methods.” From this perspective, studying AI requires methodological approaches capable of encircling these systems through experimentation and reflexive engagement. Such approaches make visible the epistemological assumptions and power relations embedded in contemporary AI infrastructures and practices of knowledge production.