Click the star to add/remove an item to/from your individual schedule.
You need to be logged in to avail of this functionality.
Log in
- Convenors:
-
Fiorenzo Polito
(LAMA Social enterprise)
Michele Mosca (Human Foundation)
Send message to Convenors
- Format:
- Paper panel
- Stream:
- Methods - research, participation and practice
Short Abstract:
The panel explores how evaluation and impact assessment can be reimagined in response to global crises, focusing on participatory, bottom-up approaches that empower communities, challenge power imbalances and promote collective action in the global North and South
Description:
Evaluation and impact assessment are crucial in development, revealing outcomes and pathways for social change. However, in an era of crisis - including the genocide in Gaza - the role of evaluation is under increasing scrutiny. Traditional top-down methods, often driven by donor priorities, positivist metrics and results-based management, can reduce social change to what can be counted, perpetuate power imbalances, marginalise community voices and limit opportunities for transformative change.
This panel explores how evaluation can be reimagined in response to uncertainty, and how evaluation practices can contribute to new forms of empowerment, justice and collective action in both the global North and South. While evaluation remains a tool for assessing the outcomes of interventions, its potential to support community-led transformation is under-exploited. Participatory and bottom-up approaches are therefore essential, challenging assumptions about assumed development and progress.
This panel invites papers focussing particularly, but not exclusively, on the following questions:
- Can evaluation practices, while ensuring accountability, become a site of resistance and opportunity for communities, and how?
- How can evaluation be a space for generating hope, collective agency, and alternative visions of the future?
- What are the constraints, inconsistencies, contradictions in democratising evaluation, and what steps can be taken to address them?
- What is the role and positionality of practitioners in ensuring accountability to communities and what are their responsibilities in this process?
We welcome theoretical and empirical contributions at different stages of development from academics, practitioners and stakeholders reflecting and working on these issues.
Accepted papers:
Paper short abstract:
Large-scale forest policy impact evaluations face challenges: time constraints, sensitive findings, and equity concerns. Drawing on case studies from Indonesia, this study explores actionable recommendations for timely, equitable, and politically aware approaches.
Paper long abstract:
Large-scale impact evaluations of forest policies are critical for assessing whether enacted interventions achieve their intended outcomes, such as reducing deforestation, promoting restoration, or alleviating poverty. Recent advancements in data quality, availability, and methodologies have enhanced these evaluations. However, significant challenges remain. First, time constraints pose a major hurdle. Policy evaluations require sufficient time for implementation and measurable effects to emerge, delaying the availability of reliable insights. Data sources like national surveys, censuses, or satellite imagery often take years to become available, and preparing them for analysis is time-intensive. Yet, the urgency of the climate crisis and global commitments (e.g., the 30x30 restoration target) pressure policymakers to demand rapid results. Second, the sensitivity of evaluation findings can create obstacles. Results—particularly negative ones, such as evidence of no impact, increased deforestation, or heightened marginalization—may face political or financial pushback due to implications for donors or stakeholders. Finally, justice-related concerns arise. Large-scale evaluations relying on secondary data often prioritize efficiency over equity, risking the marginalization of forest-proximate communities by reducing their lived experiences to mere "data points." This study delves into these challenges in detail, drawing on ongoing impact evaluation case studies in Indonesia. It also explores actionable recommendations to address these issues, emphasizing the need for timely, equitable, and politically aware approaches to forest policy evaluation. By doing so, it aims to contribute to more effective and just forest governance in the face of the polycrisis we face today.
Paper short abstract:
This paper applies a mixed-methods approach to evaluate Indonesia’s social forestry programme. Combining quantitative quasi-experimental methods with qualitative realist synthesis, we highlight challenges and strengths of integrating methods, and its value at the science-policy interface.
Paper long abstract:
Current impact evaluations designed to inform policies for better environmental and social outcomes often focus on quantifying the effects of policy interventions. However, qualitative methods—despite their value in systematically exploring causal processes and addressing the perspectives and underlying assumptions of different actors—are less commonly integrated into evaluation designs. Combining quantitative and qualitative methods in impact evaluations offers a broader and more diverse understanding of impacts, enriching the evidence base and enhancing relevance at the science-policy interface.
In our paper, we reflect on the application of a mixed-methods approach to assess the impact of Indonesia’s social forestry programme-a nation-wide initiative aiming to improve the sustainable management of forests, alleviate poverty, and prevent land conflicts. Quantitative methods are used to measure the impact of social forestry on deforestation and the well-being of local communities. We combine this method with a realist synthesis method to address the mechanisms through which social forestry is contributing, or hindering, its intended goals, and further investigate the context through which initiatives succeed or fail, and for whom.
We discuss some of the challenges of integrating qualitative and quantitative methods, particularly the paradigmatic tensions; and the difficulty of reconciling spatial scales when evaluating impacts. Furthermore, we highlight the strengths of a mixed methods approach in foregrounding the importance of context, rigorously interrogating the question of why an intervention has or hasn’t worked and moving beyond the narrow focus on measuring ‘how much’.
Paper short abstract:
Impact Evaluation in North-South vs. South-North relations describe a process of Power Shift where all stakeholders are involved in a negotiation process, taking into account different and conflicting interests, within a decolonization dynamics promoting community empowerment and reframing of roles.
Paper long abstract:
This paper focuses on the Lab-Com experience of social change and empowerment in “real” communities worldwide, namely in fragile contexts in times of crisis.
While “traditional” evaluation approaches favour the role of external consultants for the sake of impartiality, objectivity, accountability, applying rigorous rules and procedures, deepening unbalanced relations between donors and development organisations and between development organisations and communities/people, impact evaluation adopts a participatory and decolonising frame, promoting equity and redefinition of roles, focusing on power shift rather than measurability.
According to the Theory of Change approach, development interventions facilitate processes rather than mere results, empowering communities and people to take the stage with their own interests and conflicting positions: adopting an impact evaluation frame the assessors are more facilitators than experts, valuing change more than respect of initial plans and pacts: a successful community development project generates new plans and pacts, new dynamics, so that what was planned in the design phase cannot be positively evaluated if it has not evolved into something different.
Impact evaluation tells new stories and reframes the contexts, thus generating new needs and hope in continuos changes and achievements. The paradox being that the evaluation step opens new paths, communities and people involved are less satisfied because new perspectives and expectations emerge, contradictions are raised rather than appeased.
Examples form the community of practitioners involved in these processes include action-research, equity evaluation, participatory regranting mechanisms, community score cards assessment, collaborative practices and transformative approach requiring a mindset change in national and international contexts.
Paper short abstract:
This research explores how multiple accountabilities for international and sustainable development are created in three phases: discursive level, enforcing level, and implementation level through a World Bank participatory development project.
Paper long abstract:
This research explores how accountability is created under the encountering of global development discourses and institutions in the local context within a developing country, implementing a participative development project. However, accountability cannot operate only with the discursive influence and through such institutional involvements. Instead, the local context in which such a discourse encounters must be incorporated through various types of social, cultural, political and technological mechanisms, and accountability construction is, therefore, social, political, cultural, and discursive. To comprehend its impact, this study selects a World Bank development project implemented by the Sri Lankan government, which promotes sustainable agriculture and irrigation in farming communities which possess 2000-year-old traditional knowledge of agriculture and irrigation systems.
Accordingly, this research explores the multiple accountabilities for international and sustainable development in three phases: development of participative projects at the discursive level (World Bank), incorporation of local agencies at the enforcing level (ministries and government officials), development project on the ground/ grassroots at the implementation level (community organisation and empowerment). I argue that the intellectual space where accountability practices in developing countries can be located as ‘development’ in this study, and I define development as a form of accountability.
I chose the ethnographic approach, and the study analyses data collected using observations of the farming community and interviews with farmers, government officials and the World Bank officials and documentary reviews with a social theory (assemblage theory). This study will contribute to the interdisciplinary literature on accounting, accountability and development studies.
Paper short abstract:
How my relationship with evaluation reports transformed over the years. Initially, there was a lot to catch up on: Log frame, theory of change etc. which shaped these efforts. I now look at these assignments as an opportunity for my interlocutors to voice their problems.
Paper long abstract:
The paper will discuss how over the years as an occasional development project reviewer I have evolved as an evaluator and how it changed my relation with the task. My early forays into project evaluation were quite by chance; as a young lecturer, I was part of a group evaluation team of seasoned people in the development field. As a new entrant in the field, there was a lot to catch up on: Log frame, theory of change etc. My earlier efforts were often shaped by these tools and often not in my choosing. However, the more I started to work on these projects, I soon found myself having a changed relation to these tools and the very nature of my task. I increasingly began to think of these evaluation reports as opportunities for my interlocutors (often the project participants) to voice their problems. Lately, I have realized that as an evaluator, a term not to my liking, the role is not about policing the organization but talking about it and its work with some kind of caring vigilance. By emphasizing what the interlocutors think about the project and its achievement, I can talk not only about the "achievement" (for which the organization under review and its staff can often have some pressure) but also about where it fails and what can be done in the future in terms of approach. A futuristic outlook often makes this evaluation report writing experience interesting and worthwhile experience.
Paper long abstract:
In Italy, the Fund for the fight against educational poverty was established in 2016, active throughout the country. The management of the projects financed by the Fund was entrusted to partnerships of non-profit and public organizations, selected through calls for proposals. Universities and research institutions were involved in the evaluation of the interventions.
Social researchers, therefore, were responsible for designing and implementing impact evaluation plans for complex projects, differentiated by objectives and interventions, by reference territories and size, although attributable to the frameworks defined in the project calls for proposals. The impact evaluation made it possible to come into contact with a plurality of actors and beneficiaries (children, families, communities) and to question the effectiveness of the interventions, sustainability and capacity to activate local resources. Furthermore, it was necessary to redefine methodological approaches and research tools to follow the physiological and exceptional adaptations (e.g. the pandemic) of the projects, and to create opportunities for reporting the evaluation results.
This contribution proposes a reflection on the public responsibility assumed by social researchers towards project partnerships and beneficiaries, as well as towards the Fund's management body (Social Enterprise With Children). The aim is to give back the learnings that can be exploited for future planning of actions against educational poverty.
The knowledge presented derives from the debate that originates from the collection of some evaluation experiences and from the discussion activated in seminars between Universities and Research Institutes within the Observatory on social services and poverty.
Paper short abstract:
UK aid programmes like the multimillion Girls Education Challenge rely on private contractors to provide monitoring and evaluation. A critical political economy analysis shows how such approaches reinforce north-south power imbalances whilst also illuminating opportunities to enhance localisation.
Paper long abstract:
The Girls Education Challenge (GEC) was a long-running, multimillion pound UK aid fund dedicated to the provision of quality education and learning for over 1 million marginalised girls. A consortium of largely UK private sector actors were responsible for the financial management, and the monitoring and evaluation (M&E) of GEC programmes implemented in 17 countries. The paper draws on interviews with implementing organisations in several African countries, conducted as part of a 3.5 year research programme on the role of for-profit consultants and contractors in UK aid.
A critical political economy perspective allows for the analysis of north-south and public-private dynamics in the GEC M&E consortia model. The analysis shows how the design and delivery of M&E processes in the GEC were dominated by top-down approaches. The findings raised concerns about the extent to which M&E processes were excessive, undermined programme delivery, and reinforced knowledge and expertise about ‘what works’ in aid for education among contractors in the global north, risking further disempowerment of development actors in the global south.
However, there was also evidence that local actors were emboldened by their experience of M&E processes in the GEC. Some argued that rigid financial monitoring should be a vehicle for increasing trust – and financial flows - between donors and local implementers. Here, the critical political economy perspective serves as a point of departure for discussing whether and how M&E might be a vehicle for enhancing the autonomy and resourcing of local actors through aid for education and skills.
Paper short abstract:
The paper presents a case study of a project where, instead of considering the significant challenges that have slowed down the schedule and partially redefined the expected results as failures, the evaluation worked as an opportunity to clarify and support ongoing transformations
Paper long abstract:
The ongoing evaluation of development cooperation projects is generally oriented towards verifying the progress of projects against the initial logical framework and identifying strategies adopted to address emerging challenges. Little space is left for exploring emerging systemic perverse effects or development possibilities initially not foreseen by the project.
This paper presents the mid-term evaluation results of a project funded by the AICS in Tunisia. The project aims to ensure quality, equitable, and inclusive education and promote access to educational pathways for young people and adults with disabilities.
Contrary to what was initially planned during the design phase, some significant external challenges (misalignment in the level of engagement of the competent ministries, informality of the local labor market, resistance from the families of people with disabilities) and internal challenges (changes in administrative procedures, turnover of key project figures) have slowed down the schedule and partially redefined the expected results. In traditional evaluative logic, these transformations would be read as failures. Still, in this case, mid-term evaluation worked as a tool and an opportunity to clarify and support ongoing transformations.
The evaluative approach adopted aimed to provide useful information for the reflective accompaniment of the final stages of project implementation. It highlighted risks and threats while supporting the actors involved in implementing the intervention and encouraging the capitalization of innovations (organizational and systemic) that have occurred in the project's reference contexts.
Paper short abstract:
Mental models are a way to conceptualize worldviews at the individual and collective levels in culturally responsive evaluation (CRE) processes. This paper introduces a way to playfully explore possibilities of the past, present, and future through imagining worlds we want to exist into reality.
Paper long abstract:
Program evaluation is a field that can be considered either values-free or one solidly values-driven. Mental models are meant to offer a glimpse into the perspective how someone views the world through aspects like their personal values, training (e.g. education, disciplinary, or methodological), and contextual and political factors (Greene, 2007). The first step in understanding and creating a shared mental model involves understanding mental models at the individual-level. Individual mental models can be used for reflective discourse on connections in differences. Connections are then used to build a shared mental model for a group.
As an evaluator grounded in values-driven evaluation, specifically culturally responsive evaluation (CRE), this paper uses play for the imagining of symbolic worlds through the lens of realistic utopia dreaming. This activity is one way to think about Playful CRE, a way to use play as a method for disrupting loops of reoccurring time for changing the past and present for imagining new futures into reality within the CRE process. This Playful CRE activity shows a playful way to facilitate mental model (individual and collective) processes using ideas of imagination and dreaming. Activity participants are led through the process of imagining both their individual and collective evaluation utopias. Connecting a reimagined individual and shared mental model process with the engaging stakeholder stage of the CRE process offers an opportunity for the evaluator to build trust through transparency and authentic playful engagement (Hood et. al, 2015; Mathie and Greene, 1997).