Short Abstract

This presentation focuses on the ‘Design Test’ of Building Choices, a new HMPPS Accredited Offending Behaviour Programme, rolling out nationally across Prison and Probation. It demonstrates how a test and learn approach can support policy and programme design through early, embedded evaluation.

Description

Building Choices is a new HMPPS Accredited Offending Behaviour Programme, which is rolling out nationally in 2025 across Prison and Probation. The overarching strategy involves three distinct evaluation/ programme development phases: a test phase, a national rollout with a Randomised Control Trial and theory-based evaluation, and a final impact evaluation using quasi-experimental designs. This presentation focuses on the first phase, the ‘Design Test’, demonstrating how a test and learn approach can support policy and programme design through early, embedded evaluation.

Historically, Accredited programmes have been developed using a more traditional ‘waterfall’ policy making process, with implementation decisions made on the best available evidence at the time. Data on the effectiveness of a programme has typically become available long after the programme has rolled out; sometimes results have proved promising (Brinn et al. 2023), other times disappointing (see Mews et al. 2017). The Building Choices team sought to address this by bringing forward the learning process, and creating dynamic feedback loops between policy, operational and evaluation stakeholders. Central to this approach was a theory-based, Realist-informed evaluation framework testing critical assumptions and assessing evidence for key mechanisms of change at the outset.

The Building Choices Design Test brought together a multi-disciplinary team to develop and test the programme, prior to ‘accreditation’. To receive accreditation HMPPS must submit evidence to the Correctional Services Advice and Accreditation Panel (CSAAP) - an independent panel of international experts who review programmes against a set of criteria, drawn from the principles of effective interventions. This created a way of formalising and building in a test and learn culture into the governance of the programme.

The Design Test of Building Choices involved a small pilot with 3 prisons and 2 probation areas delivering the programme. A collaboratively developed theory of change created a structure to bring together evaluation, design, assessment and quality assurance experts. It highlighted key assumptions such as the effectiveness of the assessment process and the appropriateness of the programme design for those with a range of criminogenic needs. Evidence on whether these assumptions held was then assessed using qualitative research and monitoring data. Feedback loops allowed finding to be communicated and acted upon during the programme development phase.

It also brought forward the mechanisms of change underlying the programme. Areas targeted through the cognitive behavioural approach of the programme such as being better able to manage thinking, having better insight into emotions and greater openness to learning new skills could be assessed through pre/ post outcomes data as well as qualitative research. Triangulation of qualitative evidence, quantitative monitoring and outcomes data and quality assurance data generated a robust understanding of the evidence against this theory of change.

The Design Test has provided a rich and robust understanding of whether the programme is showing promise at an early stage in the development cycle. The test and learn approach enabled ongoing and iterative programme development, as well as building an evaluation culture. As a result, HMPPS is now well-positioned to implement the programme on a broader scale.