Send message to Author

Short abstract:

This paper offers a computational case study of the relationships between how heath AI is measured and promotional language use in biomedical reporting. The results offer insights into discovery-justification interactions and can inform prospective approaches to addressing hype in health AI.

Long abstract:

Hyperbole in biomedical reporting has driven an overly enthusiastic embrace of these AI technologies. For example, premature of adoption of the Epic sepsis model has been linked to diminished outcomes in hospitals that predominantly serve marginalized populations. While there has been much discussion of hyperbole as a framework for addressing these issues, dominant conceptions of hyperbole rely on problematic notions of correspondence epistemology and discovery/justification extricability. This leads to research designs that evaluate hype retrospectively, assessing various correspondences such as the fit between reporting enthusiasm and underlying AI performance or mismatches between causal language and conducted statistical tests. Given the manifest harms of AI hype, a prospective orientation is necessary. To that end, this study offers an extended computational case study of promotional language in health AI. A supervised machine learning model was used to identify promotional language in a random sample of 1200 health AI abstracts drawn from PubMed, and quasi-Poisson regression was used to assess the relationship between performance metric selection and promotional language frequency. The results indicate that the use of certain computer science metrics predicts higher rates of promotional language when compared to more traditional biomedical measures. The computational case method here offers aggregate-level insights into discovery-justification interactions, and in so doing provides a foundation for future research supporting prospective approaches to addressing hype in health AI. Increased knowledge of which methods and metrics are more likely to lead to hype in health AI can provide an evidence-based foundation for early intervention.