Send message to Authors

Paper short abstract

We examine how social roles influence evaluations of the trade-off between explainability and accuracy in AI-based health care. We find that participants adopting the perspective of an affected patient rather than a neutral observer exhibit stronger preferences for accuracy over explainability.

Paper long abstract

Artificial intelligence (AI) systems are increasingly adopted in healthcare. Many of the most accurate models operate as “black boxes”. This creates a tension between maximizing predictive performance and ensuring explainability. Explainability is emphasized as a prerequisite for patient autonomy because it ensures transparency and human oversight in high-risk applications. However, research shows that laypeople value interpretability of AI but are willing to sacrifice it for accuracy, especially in high-stakes contexts and health care settings. This raises the question of whether patient autonomy in the design of AI systems can be pursued solely through a focus on explainability. In an online survey experiment, we investigate whether people’s evaluations of the trade-off with accuracy differ when they consider it from the perspective of (1) a neutral observer, (2) an affected patient, or (3) a physician using AI-based risk assessments. We find that participants who adopt the perspective of an affected patient exhibit stronger preferences for accuracy over explainability of an AI system than neutral observers. Our findings suggest that individuals’ willingness to prioritize explainability over accuracy may depend strongly on how they are affected by the AI system’s assessments. An empathy gap may exist between the actors that lead societal discussions about AI systems (i.e., those not directly affected) and affected patients. This raises concerns about value-based patient autonomy. Clinical AI systems may need to be tailored to ensure that both transparency and patient values are meaningfully balanced and integrated into AI-mediated risk assessments.