Short Abstract

Robustly assessing how government funding achieves policy goals remains a challenge. This paper synthesises our experience of using Contribution Analysis (CA) in energy and environment evaluations. Drawing on practitioner insights, we explore the challenges and lessons learned through using CA.

Description

This paper synthesises learnings from recent evaluations in the energy space that have applied Contribution Analysis (CA) to assess the contribution of UK government interventions towards net-zero policy goals across projects, programmes, and market mechanisms. For example, this includes evaluations of innovation funding schemes (such as Heat Pump Ready), retrofit funding schemes (such as the Social Housing Decarbonisation Fund), and market interventions (such as the Capacity Market). Drawing on practical experience in delivering evaluations with a public policy consultancy, we explore how Contribution Analysis, often combined with Process Tracing (PT) and/or informed by the work of Delahais and Toulemonde, has been delivered in live evaluation contexts.

Our synthesis highlights methodological challenges encountered when applying CA in live policy environments. Specifically, we critically assess the use of CA as tool to assess the contribution of interventions when evaluation projects specify the use of CA but it is not wholly fit for purpose: 1) complex and interlinked multi-programme evaluations, 2) evaluations with poor quality data sources, 3) evaluations using CA in parallel with programme delivery where there is insufficient time for contribution to be clear, and 4) smaller scale evaluations without resources to collect sufficient data to exploit the value of CA’s evaluative power.

We examine how we as evaluators have navigated these challenges to produce credible conclusions on contribution stories that have informed decision-making, and how this has refined our approach to theory-based evaluation to ensure impact for policymakers. The analysis identifies lessons from our experience on how to best utilise Contribution Analysis, Process Tracing and the work of Delahais and Toulemonde in order to develop and test compelling theory based contribution narratives. Namely, which programme objectives are met, how market mechanisms adequately incentivise participants and whether project teams successfully deliver innovation through their grant funded projects.

By combining the power of Contribution Analysis, Process Tracing, and the approach of Delahais and Toulemonde, our frameworks have evolved to allow us to deliver two elements in parallel. Robust assessments of whether an intervention is necessary and/or sufficient to achieve its objectives, as well as a clear commentary on the strength of the evidence used to form those judgements

Beyond our methodological insights, this paper also demonstrates how CA can bridge the gap between evidence and action in the energy sector by clarifying the role of government funding in achieving policy outcomes. We identify where CA has, and has not, effectively leveraged insights for future policy design and the reasons for these successes or failures.

We argue that CA’s structured approach to causal inference strengthens accountability. By linking the use of CA with PT and elements of the work of Delehais and Toulemonde, programmes can be robustly assessed against their original intention, rather than their actual out-turn.

By sharing practitioner perspectives and cross-case lessons centred on the energy sector, this session will contribute to the conference themes of influencing policy and programme change, building evaluation cultures, and communicating evaluation for action.