Paper short abstract:

Unlike other kinds of speakers for whom verbal disfluencies are seen as evidence of being morally or cognitively deficient, chatbots seem to project a sense of subjective depth for some users because of - and not in spite of - the fact that they are extremely flawed interactional partners.

Paper long abstract:

Large language models excel in the reproduction of genres and generic texts (Gershon 2023), yet nevertheless frequently produce semantically incoherent ones. This asymmetry between the chatbots' incredible facility for syntax, genre, and combination coupled with their lack of human-like semantics has created a recurring dynamic in which that partial incoherence seems to encourage some users to search for hidden personas underneath or within the chatbot. That is, the bots’ facility with structural cotextuality (Silverstein 1997) and yet lack of contextually-deployed sense categories has left some users searching for a “real” speaking voice. This asymmetry in capacities is exploited in what people refer to as jailbreaking chatbots, in which users try to prompt the chatbots into revealing a “truer” persona by circumventing some of the safeguards built into the bots by their engineers. The search for subjectivity in one’s speaking partner has a much longer history than just the few years or months that people have been interacting with chatbots (see Peters 1999). Using arguments from religious studies (Johnson 2021), I argue that the search for a true chatbot persona is just the technologized form of a much broader search for a hidden agent, as when participants in religious events try to uncover the soul inside a subject or the god possessing a speaker. The trick with chatbots is how quickly and easily they seem to project that sense of subjective depth because of - and not in spite of - the fact that they are extremely flawed interactional partners.