Short Abstract

Lessons from the ICO’s impact reporting of an ambitious and fast-paced project focused on advertising cookies. We will showcase a theory-based approach that can be deployed in any situation to generate real-time actionable insights from evaluation findings as well as post-project reporting.

Description

How do we make insights from regulatory impact measurement as appealing and easy to digest as a chocolate chip cookie? The Information Commissioner’s Office (ICO) initiated the ‘Cookie banner project’ to improve compliance in the online advertising industry, analysing the cookie banners of the 1,000 most popular websites in the UK. We, the ICO’s Impact and Evaluation team, were tasked with writing a recipe (dropping the cookie metaphor now…) for measuring and reporting on the project’s impact.

The session will walk you through our approach from collaborative theory of change design to influencing decision-makers. We will cover the lessons we learned and the tools we implemented to provide actionable insights that can be applied by any evaluation team, including:

- Combining tools to enable visual story telling and clear reporting: where audiences have varying degrees of experience, it is important to have a varied toolkit ready for engaging with them. We will show how tools like interactive whiteboards, dashboards, and data entry platforms can be combined with an organisation’s existing board reporting arrangements to catalyse the use of insights.

- Delivering real-time insights early on to gain buy-in: winning project delivery team members over early on pays dividends when it comes to drawing on their time for evidence collection later. Our approach involved delivering small quick wins to colleagues at all levels, providing them with insights and time-saving tools to improve buy-in. This included reporting tools, data input tools, automated processes, feedback mechanisms and advice and guidance on case-making and communications. It was often more about demonstrating how common evaluation tools could also be drawn on to inform project delivery than it was about designing whole new processes and products to meet their needs.

- Theory of change (ToC) for the masses: Getting colleagues to engage with and own the project’s ToC requires involvement throughout the process. At inception, we set up design workshops using whiteboards so that colleagues could take a hands-on approach to shaping the ToC. We then layered a real-time dashboard on top of the ToC to bring the theory-based evaluation to life, demonstrating how activities, outputs and outcomes were being delivered as the project progressed. This, coupled with our organisation-wide theory of change training initiative, brought about a step-change in outcomes and impact-based decision-making for the project.

We would also love to hear from you if you have experience using any of the tools and approaches we cover during the session, or any alternatives. We welcome engagement during the Q&A and after the session so that we can learn from your experiences.