Short abstract:

Platform companies rely on Trust and Safety teams to advance “good” behavior. This ethnographic study reveals that T&S professionals navigate conflicts between achieving “good” and their company’s business goals and organizational dynamics, structurally conditioned by the nature of these companies.

Long abstract:

Platform companies make decisions surrounding what user-generated activity is admissible. The work of defining, preventing, and stopping the abuse and misuse of platforms is tasked to “Trust and Safety” (T&S) teams. These teams act against phenomena like “fraud,” “account takeovers,” “misinformation,” or “spam”. The design of these strategies, including policymaking and operationalization, relies on normative ideas of “good” behavior. In this paper I ask: how do Trust and Safety professionals work towards the goal of “good?”

To answer this, I conducted an ethnographic study of the T&S field, including interviews with 30 T&S professionals. I find that T&S professionals engage in constant “persuasive work,” aiming to connect the normative goals of “good” to the incentives, designs and limitations of platforms companies. Often times, T&S professionals perceive their normative goals in contradiction with the business aims of the company, like user growth or profit. To address this, they embrace the language of “risk” and engage with stakeholders throughout the company. Furthermore, T&S workers also see their normative goals challenged by the organizational dynamics of the firms they work at. As many platform companies are publicly traded, leadership often responds to shareholder pressure or anticipates shareholder behavior. This leads to frequent changes in organizational strategy and design, including layoff and “re-orgs,” which limit the perceived success of these normative strategies. Ultimately, my work shows that the achievement of normative goals of “good” by contemporary platforms are structurally conditioned by the incentives, designs and limitations of platforms companies.