Innovation Across Borders: Kazakhstan
José Sierra Castillo, UNICEF Europe and Central Asia, tells us about his experience testing a machine learning model for policy design in Kazakhstan
Across 190 countries and territories, UNICEF colleagues and partners are on the frontlines of the greatest challenges affecting the lives of children and young people. Innovation Across Borders highlights the experiences, successes and learnings of innovation champions, committed to making positive social impact.
What is causal machine learning and what problem does it help address?
Causal machine learning is a relatively new area of artificial intelligence (AI) that shifts the focus of data analysis from prediction to explanation, from correlation to causation. The implications for policy design are powerful.
Every policymaker wants to ensure their programme has the greatest possible impact on a target population, but they often lack the right tools for the job. Randomized controlled trials are the current gold standard for measuring impact, but their deep analytical rigour is matched by their high cost, and the results assess averages not individuals. For example, an early childhood development programme might appear to have no effect on the average child, while having an enormous impact on a specific subset of children.
Averages hide individuals, but causal machine learning identifies what works and for whom. That same policymaker can see that their early childhood development programme in fact has an important effect on, say, children living in poverty in their country’s rural north, even if, on average, the programme’s benefits are less clear.
You tested this in partnership with the government of Kazakhstan. How did it go?
Working with the new Ministry of AI and Digital Development was an enriching experience that showed me how government partners can be deeply engaged in innovation, and how easy it is to implement new ideas when everyone is invested in them.
The government in Kazakhstan was eager to analyse the impact of its household cash transfer programme so we worked with a team from the ministry, in consultation with the Ministry of Labor and Social Protection, to build a causal machine learning model that could explore the programme’s effect on different groups. We found that, on average, the programme had a minimal positive impact such that a cost-benefit analysis might suggest the money could be better spent elsewhere. However, our algorithm discovered that the most vulnerable members of society benefited from the programme at a rate two-to-three times higher than the average recipient.
Our pilot showed that causal machine learning can help governments tailor interventions to increase the impact, reach and cost efficiency of their programmes.
What challenges did you encounter during this project?
As we scaled and fine-tuned the model, we had to iterate towards improved data quality, move from a fully data science team to a multidisciplinary one and increase computing power to train the model. We were actually surprised by how easy it was to train the model. This is because, while causal machine learning is still an incipient field, there are nevertheless simple, easy-to-transfer, open-source tools available. The main challenge was the subject matter expertise required to validate the model outputs. We realized early on that this kind of work requires an interdisciplinary approach.
What’s next? Is the Kazakhstan government able to use the technology day-to-day?
With funding from the UNICEF Venture Fund, this proof of concept was tested on around 600,000 households, and the government is now working on scaling the model nationally. The Ministry of Labor and Social inclusion is working closely with the Ministry of AI and Digital Development to guarantee the scale up produces relevant and applicable findings. They continue to iterate on data quality; work with cash transfer experts to advise on the theoretical aspects of the model; and secure sufficient computing power to train the model on all of Kazakhstan’s 6 million households. After that, the technology will be tested in other policy areas, such as education and health.
What advice do you have for partner governments that wish to adopt causal machine learning for policy design?
Don’t let perfect be the enemy of good. Our first instinct might be to wait for just the right data before training the model, but the speed at which AI is changing and evolving means this isn’t necessary. Instead, we can build a test environment and work with what data we have. The training of the algorithm reveals data gaps, which lead to refinements of the algorithm, which in turn reveal more data issues, leading to further refinements, and so on. It’s an iterative process – getting started is the key.
