Short Abstract

How can evaluators justify methodological choices when facing competing views of "good" evidence? We present the Principled Adequacy for Purpose framework, a practical approach for navigating this tension and determining which methods are fit-for-purpose given our aims, values and context.

Description

How do evaluators justify method choices when stakeholders hold different views about what constitutes "good" evidence? Whether explaining why an RCT is essential for policy influence, defending qualitative approaches for understanding mechanisms, or advocating for participatory methods, evaluators need principled ways to demonstrate that their methods are fit-for-purpose. This talk proposes a novel 'Principled Adequacy for Purpose' (PAP) framework for this, aiming to bridge evaluation design into actionable evidence. Centrally, we argue that methods are fit-for-purpose if and only if using them is likely to achieve our evaluative aim(s), relative to our context, including practical constraints, questions and values.

Reviewing previous work on methodological choice and justification, we identify three strands: evidence hierarchies (critically examined eg by Cartwright, 2007), questions focused frameworks (eg Stern et al., 2012, or Befani, 2020), and more recently, value focused frameworks (Aston et al., 2022, Apgar et al. 2024). Critically, these approaches share two limitations. First, they pay insufficient attention to the value of evaluative purposes, as explored in use-focused evaluation theory. Second, they fail to offer practical and complete guidance, putting undue burden on decision makers or proposing methods that may still fail to achieve our purpose.

PAP addresses both limitations. Drawing on recent work on the concept of 'Adequacy for Purpose' (Parker, 2009), we argue that reintroducing purpose resolves these practical and theoretical limitations. We show how question and values approaches are fundamentally complementary, each identifying dimensions of potential inadequacy. For example, our approach underscores that an RCT answering the right question may still fail if it cannot enable stakeholder learning; conversely, a participatory method aligned with community values fails if it cannot achieve the required levels of accountability.

The presentation highlights four key benefits of PAP. First, it acknowledges methodological uncertainty: methods are "likely" rather than guaranteed to achieve purposes. Second, it creates space for methodological diversity, with multiple approaches possibly adequate for given purposes. Third, it provides defensible justification for appropriate methods, whether experimental, qualitative of participatory, if these adequate-for-purpose. Fourth, it clarifies how values legitimately influence choices without compromising validity. At the same time, we also acknowledge PAP's limitation: deeply entrenched preferences may resist change; the framework requires principled application to avoid post-hoc rationalisation; and determining the "right" purpose can remain challenging. Yet for evaluators navigating diverse stakeholder expectations, PAP provides theoretically grounded, practically applicable justification for methodological choices that can drive action.