Short Abstract

Outcome Harvesting is powerful but hard to scale. We give an example of an evaluation which used an AI interviewer to conduct adaptive OH conversations. We discuss ethics, privacy and validation safeguards and the potential for scaling OH data collection.

Description

Outcome Harvesting (OH) excels at surfacing behavioural outcomes and understanding their significance and contribution. Its weakness is scale: high‑quality OH relies on iterative, probing interviews with many stakeholders - hard to deliver when hundreds of voices matter and timing is tight. This talk presents an OH evaluation which explored OH interviewing using an AI interviewer on the QualiaInterviews platform, while preserving transparency and quality.

The case study was an OH evaluation of an Austrian climate‑adaptation pilot (KLAR!). A detailed instruction was developed for the AI interviewer to enable it to carry out OH-style interviewing. Interviewees were invited by sending them a link to the interview which they could complete in their own time. The AI was told to conduct adaptive multi‑turn conversations that branch, circle back and generate sub‑questions to elicit outcome, significance and contribution statements. It was also instructed to make a summary or the interview at the end to make the process more transparent and to make an immediate contribution to the verification process.

The interviews generated a rich set of outcome leads: several met OH SMART criteria outright; others became leads for further verification. It was judged to be particularly useful for the "first round" of interviews to increase reach, stakeholder diversity and inclusiveness from the start - as a basis for priorisation of outcome leads, to focus on certain outcome domains or high-priority outcomes. More work was still necessary from the human evaluators to continue and complete the OH process, including conducting a smaller number of follow-up validation interviews.

Real-time summaries created by the AI interviewer during the interview increased transparency and gave participants ownership of how their contributions were being recorded, something that can be difficult for human interviewers to do while also concentrating on the interview itself.

We also report our experiences with AI-supported analysis of the resulting transcripts, including extraction of Outcome Statements (outcome, significance, contribution), working both on a per-interview basis and also globally. AI support was useful in this but human guidance and validation were still essential.

Writing detailed OH interview instructions is not easy, so we close with pragmatic guidance: writing robust interview instructions; enabling multilingual access; privacy options (including EU‑only data residency); limitations/mitigations (bias, hallucination, consent, and oversight). Participants leave with a better understanding of the potential of, but also challenges with, AI-supported OH interviewing.

This contribution directly supports UKES’s sub-theme 2 on embedding evaluation into daily practice, showing one way to relatively easily include a very wide set of stakeholders in the Outcome Harvesting process, and also showing one way to make use AI in a relatively transparent way.