Paper short abstract:

This paper has two aims. 1) Showing how the gold standard of impact evaluation for development projects, i.e., randomized controlled trials, ignores important aspects. 2) Taking inspiration from the capability approach to elaborate a richer theoretical framework aimed at informing alternative methods to assess development interventions. Health projects in Kenya are taken as a case study.

Paper long abstract:

Keywords: development projects, impact evaluation, randomized controlled trials, human development & capability approach, health

Research Context

Disadvantaged and fragile countries are historically threatened by a series of multifaceted and interlinked crises which create stress on individuals’ and communities’ capabilities, e.g., extreme poverty, humanitarian challenges, sharp inequalities, pandemics, violent conflicts, and climate change. Among these crises, one that is often forgotten is a measurement crisis concerning the assessment of development projects. Project assessment is in fact crucial for understanding the impact of actions and commitments taken for addressing these kinds of emergencies, deciding how and where to allocate the available resources, and designing future interventions. However, the methodology that currently represents the ‘gold standard’ in development economics, i.e., randomized controlled trials (RCTs), is difficult to apply in some important circumstances: a) when development programs are broad in scope, b) when they have been already implemented and thus the research is mostly retrospective, c) when their effects unfold in a medium/long-term horizon and are not limited to the short-term, d) when their implementation in a certain domain is likely to produce cross-sectorial and cross-thematic spill-overs in other domains. Since RCTs are characterized by the necessity to create a manageable experimental setting to isolate a few measurable variables, it can be tricky for them to face these challenges and achieve external validity (Achen, 2022). Moreover, other elements that bring their reliability into question are their feigning ignorance about the content of experience, common knowledge, and best practices (Pritchett, 2020) and their being guided by a rule-utilitarian inspiration indifferent to distributive concerns and detached from the adaptive preferences issue (Henderson, 2022). This paper aims to propose the capability approach as a richer informational basis for assessing development projects to address these limitations, capture important aspects often ignored by traditional evaluation instruments, and consequently orient funding allocation and project design more effectively.

Methodology

The capability approach is a conceptual framework that focuses on the evaluation of human development in terms of the genuine opportunities people have to achieve doings and beings they have reason to value in several dimensions of their lives. Hence, some scholars have argued that its constitutive multidimensional and complex character, which allows researchers to properly weigh the role of both personal and social conversion factors as well as of individual agency, and its vagueness, which transforms the evaluation of achievements from a binary issue into a matter of degree, make the capability approach particularly suitable for the assessment of contested subjects like development projects (Garcés-Velástegui, 2022). Accordingly, this approach offers several advantages for the scope of this paper: a) it provides a unifying lens through which to assess different case studies and compare projects’ real effectiveness while highlighting their specificities; b) it enables to make more detailed analyses and give priority to the worst-off, by placing individuals and communities into their respective contexts; c) it encompasses not just direct, short-term, tangible benefits, but also more long-term, indirect, intangible ones; d) it focus on the process through which outcomes are achieved rather that only on outcomes themselves.

Therefore, our paper takes inspiration from the capability approach to elaborate a theoretical framework aimed at guiding the development of alternative empirical methodologies to assess development projects by focusing on six dimensions:

1) the ownership of development projects and the partnerships that can guarantee their success;

2) the participation and level of engagement of local actors, either formal or informal, as well as the governance mechanisms implemented;

3) to what extent individual and community agency and empowerment have been activated by the projects;

4) what have been the tangible and intangible impacts generated in the short, medium, and long run;

5) what have been, if any, the cross-sectorial and cross-thematic spillover effects produced;

6) the environmental, social, and economic sustainability of the projects.

Moreover, the paper applies this framework to a specific case study, namely, the assessment of a series of development interventions that have been run in Kenya by AICS (the Italian Agency for Development Cooperation) concerning the important issues of maternal and child health and tuberculosis infection. These are among the main challenges faced by the Kenyan healthcare sector to guarantee sustainable human development to the local population and meet the targets of the 3rd SDG (“Good health and well-being”). Each project is analyzed through a mixed methodology informed by the research framework, combining retrospective quantitative surveys and qualitative in-depth interviews conducted with a purposive-selected sample of the main projects’ beneficiaries.

Analysis & Conclusion

The analysis of the projects that have been considered within the paper points to three important lessons:

1. The relevance of the health dimension for human development, both at the individual and community levels.

2. The importance of applying a multidimensional framework for effectively assessing (and designing) development projects capable of capturing the dimensions of participation and ownership, evaluating medium/long-term effects and intangible impacts in addition to short-term tangible ones, combining both individual agency and context analysis.

3. How applying the capability approach offers the opportunity to develop not a deterministic method but a holistic way of thinking that can be adapted to different cases and contexts and operationalised through different methodologies in compliance with the initial objectives of each development project.

Bibliography

Achen, C. H. 2020. RCTs versus observational research. In J. Widner, M. Woolcock & D. Ortega Nieto (eds.), The Case for Case Studies: Methods and Applications in International Development: 52–60. Cambridge, UK: Cambridge University Press.

Garcés-Velástegui, P. 2022. Using the capability approach and fuzzy set qualitative comparative analysis in development policy evaluation. Journal of Comparative Policy Analysis: Research and Practice, 24(2): 179–197.

Henderson, H. 2022. The moral foundations of impact evaluation. Journal of Human Development and Capabilities, 23(3): 425–454.

Pritchett, L. 2020. Why “feigned ignorance” is not good economics (or science generally). Lant Pritchett Personal Website, November 6. https://lantpritchett.org/why-feigned-ignorance-is-not-good-economics-or-science-generally/