Social workers in schools always boost student outcomes. Drug offenders shouldn’t be treated differently. Malaria bed nets are more likely to be used if people pay for them. Seeing inside a jail will deter juvenile delinquents from becoming criminals.
All four statements sound perfectly sensible, don’t they? Unfortunately, randomised trials suggest all four are perfectly wrong. Let me explain.
In Britain, pilots of social workers in schools showed everyone liked the idea. Teachers, social workers and students all liked it. Then researchers at Cardiff and Oxford Universities ran a two‑year randomised trial across 300 schools to test the impact. The results, reported this year, showed no significant positive impact. As a result, the planned national rollout has now been scrapped.
In NSW, a randomised trial of the Drug Court showed a tailored approach to drug offenders reduced recidivism. By treating their addiction, they became much less likely to reoffend than if they had been sentenced through the traditional criminal justice system. Drug courts don’t just help addicts ‑ they also make the streets safer.
In Africa, some economists argued free anti‑malarial bed nets would not be valued by villagers, and might be used instead as makeshift fishing nets. So a randomised trial tested the take‑up and use of free versus cheap bed nets. It turned out free bed nets were far more popular, and equally likely to be used. As the results of the randomised trials became clear, the World Health Organisation switched its policy to favour free distribution of bed nets. The results of the experiment save thousands of lives every year.
In the United States, a policy known as “Scared Straight” grew out of an Academy Award‑winning documentary. Juvenile offenders were brought into jails for a day, where they met hardened adult criminals. Low‑quality evaluations ‑ comparing those who took up the program with those who chose not to participate ‑ suggested it cut crime by up to half. But randomised trials told a different story: suggesting participating in Scared Straight made youths substantially more likely to offend.
What these four examples have in common is they used a randomised trial to evaluate the impact of a policy. Randomised trials have a long history in medicine, going back to James Lind’s randomised trials of scurvy treatments in 1747, which helped save the lives of thousands of sailors. In the 1940s, randomised trials showed antibiotics did not cure the common cold. In the 1950s, randomised trials showed the polio vaccine was safe and effective.
Randomised trials helped drive the transformation from “eminence‑based medicine” to “evidence‑based medicine”. Until the end of the late‑19th century, dangerous treatments such as bloodletting meant doctors probably killed more patients than they saved. Randomised trials helped bring a what‑works philosophy to medicine.
That’s why we announced in the 2023 budget the Albanese government will create the Australian Centre for Evaluation. A core role for the centre will be to champion high‑quality impact evaluations, such as randomised policy trials.
We’re also encouraging agencies to rebuild their own in‑house evaluation capabilities and consider partnering with the Australian Centre for Evaluation to carry out high‑quality evaluations. We will be complementing high‑quality impact evaluations with other culturally safe evaluation methods that help us to understand the lived experiences of Australians and deliver better services.
In the face of major challenges, low‑quality evaluation is a hindrance, not a help. Using flawed impact evaluation techniques is like doing your running training with a slow watch. It might make you feel like you’re fleet‑footed, but when it comes to race day, you’ll eventually be shown up.
That’s why researchers in areas such as pharmaceutical development are committed to using randomised trials. They recognise the importance of accurately evaluating new treatments. They know poor evaluation of medical treatments can cost money and lives.
In the face of hard problems, we must bring more than a crash‑or‑crash‑through mentality. We need to show up with a willingness to rigorously evaluate those solutions. We need to bring enough modesty to the task to acknowledge that answers which sound right may not always work in the real world. To generate and sustain a culture of continual learning, we need to be open to being proven wrong, and to use that information to do better the next time. We need to accept honest feedback ‑ not pretend to get by with a dodgy wristwatch.
