Paper short abstract:

We empirically examine the types of reasoning elicited by AI analyses and how they relate to EBM. We observed that while EBM advances a generalized one-in-many frequency principle, AI-based analyses offer insights into “frequent exceptions” and thus pursue a different, more diverse frequentism.

Paper long abstract:

Evidence-based medicine (EBM) relies on the ‘hierarchy of evidence’, which ranks different studies according to their methodological rigour, to assess the quality of evidence. Systematic reviews and RCTs are favoured, whereas experience-based knowledge and case studies are valued least. EBM thus evaluates evidence following a frequentist logic, i.e. drawing statistical inferences from frequent events. Despite criticism for being too narrow, inflexible and undervaluing many types of knowledge, it remains the dominant approach in EBM.

Large volumes of data, especially so-called ‘real-world data’, together with advanced data analytics present new opportunities to generate insights that are inaccessible to EBM’s core methodologies. Computational and AI-based methods are employed to process these data by identifying patterns and correlations that supposedly support the prevention, diagnosis and (personalised) treatment of diseases.

Despite the diversity of data, these AI-based technologies often operate within a frequentist framework similar to EBM, where statistical probabilities are determined by observed frequencies. AI-based analyses of ‘big data’ are thus said to reproduce prevailing modes of knowledge production in EBM that prioritize quantitative data and statistics.

Drawing on AI-based analyses of patient experiences shared online to inform evidence-based guideline development, we empirically scrutinise the types of reasonings elicited by AI analyses. While RCTs follow a generalised one-in-many frequency principle, AI methods enact a different, diverse frequentism. With AI methods, we obtain many different individual experiences, a large “gathering of exceptions”. Understanding these specific modes of reasoning illuminates the kind of knowledge that can be generated by AI and its relation to EBM.