Short Abstract

This study explores the impact of a summer learning programme implemented across Italy for students aged 9-13 from disadvantaged socio-economic backgrounds. It does so by implementing the Most Significant Change technique, a participatory, human-centred approach to evaluation.

Description

This study discusses how the use of participatory evaluation methods can be empowering for programme participants and yield more rounded findings of a programme’s impact. The study explores the impact of Arcipelago Educativo, a summer learning programme offered free of charge to students across Italy from disadvantaged socio-economic backgrounds. The programme is co-funded by Fondazione Agnelli and Save the Children Italy, to contrast ‘summer learning loss’, a phenomenon that refers to a tendency of students to forget what was learned throughout the academic year, during the summer break. Evidence shows that summer learning loss affects students from disadvantaged socio-economic backgrounds more than their wealthier peers. This issue has been further exacerbated in recent years by the school closures during the COVID-19 pandemic, widening the existing educational gap. In Italy, a country where summer breaks are amongst the longest in Europe (June to September) summer learning programmes play a crucial role in contrasting educational inequality.

Building upon the results from a previous RCT, this study implements Most Significant Change technique to learn about programme impacts from the voices of its participants. Rather than treating evaluation as an external accountability exercise, the study focused on evaluation as a co-produced learning process, engaging programme staff and children aged 9–13 as active knowledge creators through the collection of drawings and stories of change experienced during the programme. This approach provides a voice to programme stakeholders by focusing on participant-valued programme outcomes, surfacing critical insights about programme mechanisms and emergent programme outcomes.

Findings highlight three intertwined mechanism:

(1) relational safety and non-judgment, fostered by educators’ interactional work in small groups, which bolsters self-efficacy and willingness to engage; (2) peer belonging and socio-emotional regulation, cultivated through structured collaboration and guided dialogue, which reduces social anxiety, supports collaborative learning and the development of positive towards going back to school; and (3) situated, curiosity-driven learning, achieved by alternating in-class activities with outdoor and museum-based activities that connect knowledge to everyday practices, sustaining attention and positive attitudes towards learning.

The mixed-method design—an RCT on the 2022 cohort followed by MSC with creative elicitation on the 2023 cohort—illustrates how experimental measurements of whether a programme works can be significantly enriched with participatory inquiry to shed light into how it works and why certain changes are important to participants. The stories illustrate how the increase in subject test scores (measured via RCT) is not merely the result of additional practice of the summer, but rather the effect of an educational context that combines relational safety, trust, motivation and collaboration amongst peers. To the best of the author’s knowledge this is the first study implementing the MSC technique to evaluate how summer learning programmes can contribute to contrasting educational inequality. Moreover it is the first study to employ MSC to collect stories creatively in the form of drawings from young participants, demonstrating its value as a creative, participant-centred evaluation approach, easily accessible to young beneficiaries, even when language represents a barrier due to migration backgrounds and/or special educational needs.