Short Abstract

Evaluating a mega-portfolio of interventions? Trying to do it in a security sector? We’ve been there. We unpack how we tackled ill-suited criteria, abstraction, data access hurdles, and non-evaluator audiences from our work evaluating security sector portfolios, sparking debate on what credible evaluation really means at the mega-portfolio scale.

Description

We have been delivering Portfolio Monitoring, Evaluation, and Learning programmes focused on portfolios of security sector and emerging technology interventions. In one instance, this 'portfolio' is in fact a portfolio in name only, as it houses multiple sub-portfolios, all of which have multiple programmes that house projects. This is challenging evaluatively, as our role involves portfolio-level evaluations and reviews. These pieces of work must cut across a wide range of interventions, delivery bodies, and actors, all operating at different levels of society and complexity.

Delivering useful and actionable evaluation at this level of abstraction (i.e. cross-portfolio) is difficult, and security sector programming brings acute information access restrictions. Furthermore, we face the barrier of security-sector evaluation commissioners sometimes having lower evaluation familiarity, requiring very different evaluation products and decision-making support to translate evaluation into action.

In this session we aim to share our experience and prompt discussion with others evaluating mega-portfolios or otherwise evaluating the security sector. We will speak to how we have fostered evaluative practice that was practical, defendable, and useful despite operating at a mega-portfolio level in a uniquely challenging sector. We hope to show how these solutions might translate into others’ contexts.

In running this session, we will outline our context, the barriers, and how we overcame them. We wish to spark subsequent discussion with the audience on important questions facing evaluators in our position:

o Can and should government take an OECD-DAC approach to evaluation and reviews in these thematics, or when operating at a mega-portfolio level?

o How do we defensibly but flexibly look into assessing security sector topics at a portfolio-level without over-engineering new criteria that face the same problems?

o How do we define evidence, success, and credibility in reviews and evaluations that operate at a mega-portfolio level? Do you think we got it right?

o How do you evaluate for sectors newer to evaluation given the above questions?