Send message to Author

Paper short abstract

Evaluating a mega-portfolio of interventions? We’ve been there. We unpack how we tackled ill-suited criteria, abstraction, data access hurdles, and non-evaluator audiences from our work evaluating portfolios in the nexus of emerging technology and security, sparking debate on what credible evaluation really means at the mega-portfolio scale.

Paper long abstract

We have been delivering Portfolio Monitoring, Evaluation, and Learning programmes focused on portfolios at the nexus of emerging technology and security. In one instance, this 'portfolio' is in fact a portfolio in name only, as it houses multiple sub-portfolios, all of which have multiple programmes that house projects. This is challenging evaluatively, as our role involves portfolio-level evaluations and reviews. These pieces of work must cut across a wide range of interventions, delivery bodies, and actors, all operating at different levels of society and complexity.

Delivering useful and actionable evaluation at this level of abstraction (i.e. cross-portfolio) is difficult, and this type of programming brings acute data access issues. Furthermore, we face the barrier of evaluation commissioners in these sectors sometimes having lower evaluation familiarity, requiring very different evaluation products and decision-making support to translate evaluation into action.

In this session we aim to share our experience and prompt discussion with others evaluating mega-portfolios or otherwise evaluating these nexus sectors. We will speak to how we have fostered evaluative practice that was practical, defendable, and useful despite operating at a mega-portfolio level in a uniquely challenging sector. We hope to show how these solutions might translate into others’ contexts.

In running this session, we will outline our context, the barriers, and how we overcame them. We wish to spark subsequent discussion with the audience on important questions facing evaluators in our position:

o Can and should government take an OECD-DAC approach to evaluation and reviews in these thematics, or when operating at a mega-portfolio level?

o How do we defensibly but flexibly look into assessing nexus topics at a portfolio-level without over-engineering new criteria that face the same problems?

o How do we define evidence, success, and credibility in reviews and evaluations that operate at a mega-portfolio level? Do you think we got it right?

o How do you evaluate for sectors newer to evaluation given the above questions?