The Open Data Institute (ODI)’s Public Policy team is undertaking an ambitious new international project, called ‘Experimentalism and the Fourth Industrial Revolution’. We are exploring how data policymakers and data practitioners can work in more innovative and experimental ways to adapt to, and leverage, the fast-moving societal and economic challenges and opportunities around new data availability and associated digital technologies.
The project runs in three parallel workstreams named after sci fi writers. This workstream is named after Isaac Asimov and focuses on experimentation and innovation opportunities and needs for the UK as it charts a post-Brexit future.
This is part 3, which explores the evaluation and assessment stage of data policy and practice, and how innovation and experimentation here might support better questions and therefore better answers.
- Isaac Asimov part 1 – Asimov’s data revolution
- Isaac Asimov part 2 – Asimov and data revelation mechanisms
Holding up the mirror
As Isaac Asimov wrote in I, Robot (1950), “It is the obvious which is so difficult to see most of the time. People say ‘It’s as plain as the nose on your face.’ But how much of the nose on your face can you see, unless someone holds a mirror up to you?”
Monitoring, evaluation and learning (MEL) is the ‘mirror’ which we hold up to policy, and the methods and metrics we use for this can have a significant effect on policy outcomes. This is particularly influential in new policies relevant to the shaping of new data ecosystems.
On 27 June 2022, the ODI, in partnership with the Innovation Growth Lab (IGL) by Nesta and the Policy Institute at King’s College London, convened an online roundtable of international representatives from government, academia, business and civil society. Roundtable participants explored opportunities for experimental approaches to valuation, evaluation, and evidence in the context of data policy and practice. We’re sharing audio clips of the guest speakers’ provocation presentations with some discussion questions to prompt critical creative exploration of the topic.
Roundtable provocations
Introduction: Gavin Freeguard, Special Advisor, ODI
The Open Data Institute · Introduction – Gavin Freeguard, Special Adviser, ODI
Some key questions:
- How do the methods and metrics we use for data policy impact assessment shape data policy outcomes?
- How can we navigate between ‘evidence-based policy-making’ and ‘policy-based evidence-making’, where targets are engaged with in trivial ways that don’t ‘show up’ in the metrics?
- How can we balance expectations around use of granular data in policymaking, and trends towards systems level approaches to policy?
Keynote: Dr Patrick Noack, Manager, World Economic Forum Center for the 4th Industrial Revolution, Dubai; and Executive Director of Future, Foresight and Imagination at the Dubai Future Foundation
The Open Data Institute · Keynote – Dr Patrick Noack, World Economic Forum
Some key questions:
- Are policymakers asking the right questions about the purpose of digital economies?
- What would it mean to quantify aspects of data use and AI use ‘as if the future mattered’?
- How can we identify patterns and signals in the near-term that might be relevant to new kinds of longer-term impacts?
Provocation 1: Valuation – Juan Mateos-Garcia, Director of Data Analytics Practice, Nesta
The Open Data Institute · Provocation 1 – Juan Mateos-Garcia, Nesta
Some key questions:
- What are the characteristics of AI models that market-led valuations capture? And what are the characteristics that might be missed out?
- What mechanisms are available for supporting technological diversity as a public good?
- What might be the benefits or opportunities around open AI models, and what might be the challenges or risks?
Provocation 2: Evaluation – Dr Piret Tõnurist, Innovation Lead, OECD Observatory of Public Sector Innovation
The Open Data Institute · Provocation 2 – Piret Tõnurist, OECD Observatory of Public Sector Innovation
Some key questions:
- What kinds of questions are we trying to answer when we carry out evaluation in the public sector? Are these the right questions?
- What data, and what data infrastructure, could help to account for uncertainty in public sector evaluation?
- How might new technologies such as AI affect the methods or purpose of public sector evaluation?
Provocation 3: Evidence – Professor Michael Sanders, Professor of Public Policy, the Policy Institute at King’s College London
The Open Data Institute · Provocation 3 – Professor Michael Sanders, the Policy Institute at King’s College London
Some key questions:
- Is it possible to ‘move fast and break things’ in the public sector? Is it desirable?
- Are there policy areas where decision-making cannot or should not be scaled up or speeded up, even if new data available and digital technology makes scale and speed possible?
- Could AI have a role alongside traditional forms of evidence-gathering for policy, such as Randomised Control Trials (RCTs)?
Get involved
We’ve created a short summary note with a distillation of the high-level themes and observations that emerged in discussion. It’s available here as a ‘living document’, and we welcome and encourage reader comments on it, as part of a community of practice, and to inform how the project develops.
The summary note also includes a Resource Guide that we hope you find useful, and that you can contribute to. If you would like to explore any of these ideas and opportunities further with any of the event partners, or in collaboration with us, we'd be keen to hear from you. Some immediate practical opportunities include ODI research fellowships, and we are also open to proposals for collaboration on the development of case studies, projects, or activities – there's more about our relevant work in these areas in the Resource Guide. If there are projects or resources that you'd find useful but that don't seem to exist, do let us know in this document – we or others in this community of practice might be able to develop them.
Find out more about the project and sign up to the project mailing list here, contact the team at [email protected], or look out for our news on Twitter: @ODIHQ.