Policy impacts aren't a guarantee of political viability

Many scientific researchers devote themselves to finding differences between population averages. This is not less true in policy analysis. What strikes me is that all too often, the mere finding of a difference in means is taken as the end of the story, which it certainly isn't.

Consider these two populations with clearly different means:

Population A (untreated) has a mean of $2,276 and population B (treated) has a mean of $3,758. Let's imagine that these are bank account balances in thousands of dollars. You can see that the difference is not trivial ($1,481.)

However, let me ask a different question. Rather than asking if there is a difference in average, I want to know how likely it is that if I take two people at random, one from each population, the person from population B will be in fact richer than the person from population A.

I ran a simulation and the answer is close to 75%. In other words, 25% of the time, the person from population B will be poorer than the person from population A. That's a probability of 1 in 4.

Scientists don't always have the luck of bumping into such large differences in means. In our example, the mean of population B was 1.6 times that of population A. What happens if we make the difference smaller, but still acceptably large? Like, for instance, $481? This would be the plot:

The probability of a person from population B meeting a poorer person from population A would be then ~63%. In general, the more overlap there is between the two distributions, the smaller the probability of finding an actual effect if you take two people at random, one from each distribution.

And notice here that we are only checking that the bank account balances of the two individuals are different. We aren't saying anything about the magnitude of the difference. Here, the probability of the person from population B being at least $100 dollars richer is even lower (61%.)

If you want to promote a public policy solution based on an impact evaluation (which routinely compares population means,) you have to be aware of the fact that a positive impact does not mean a noticeable impact. Don't expect that people in the streets will suddenly start to realise how good the intervention was. In our second example, it is still fairly unlikely that your treated individuals will feel any richer than their untreated friends or neighbours.

My point is that even truly impactful policies don't necessarily guarantee a change in people's perceptions, and thus don't necessarily translate into political profits for policy makers. If good policies were evidently better than bad policies, then people would push for them more often. But the problem is that policy effects are not easily spotted by the general public, even if there are (positive or negative) effects. That is, unless they are extremely large. Many bad policies keep existing because their bad effects are not apparent to the public opinion, and many good policies are discarded for exactly the same reason.

***

Did you like what you read? Why not subscribe?