regression to the mean

Because program effectiveness has an element of randomness, we see a phenomenon called Regression to the Mean. In any given year, some of the programs that did well probably benefited from a dose of good luck, and some that did badly were probably hindered by bad fortune. There's no reason to expect the luck to be consistent next year, so the program which was initially lucky will probably do a bit worse, and the unlucky program a bit better.

Because bad luck makes you do worse, victims of misfortune will tend to be among the worse performing half, so when they revert to form they'll get closer to the average performance. Similarly, those benefiting from temporary good luck will be disproportionately likely to be in the top half, so when their luck fades their performance will decline. Because of this, performance over the long run tends to move towards the average (regress to the mean) as temporary perturbations fade away. Short-term high or low performers will probably be more average in the long run. Over time, charities regress towards the mean, so our estimate for how much good donating to them will do has to take this account into account.

Measurement Error

The risk of measurement error creates a similar effect. In the same way that some charities are unusually effective or ineffective in any given year because of luck, the difficulties in measuring effectiveness means that some will be over-estimated and some under-estimated. Like before, because those that are overestimated will tend to appear better and those that are underestimated will tend to appear worse, the true performance of any given charity is probably nearer to the average than the measurement would suggest.

This is a slightly different effect than regression to the mean. The measurement inaccuracy could be persistent, with the measured value never regressing to the mean if the same charities are always underestimated (or overestimated). This would mean that the measured values would never accurately depict the true cost-effectiveness. But because we fundamentally care about the true effectiveness, not the measurements, we need to correct for this as well. 

Measurement error also effects how valuable research into charity effectiveness is. If there was no measurement error, and our conclusions were always perfect, the returns of research would be extremely high. At the other extreme, if measurement errors were so large that they accounted for all the variance observed between the charities studied, the research is useless.

Modelling measurement errors as log-normally distributed, we estimate they account for around ⅓ of the overall variance in DCP's research. While this reduces the estimated cost-effectiveness of the best charities, they are still extremely cost-effective, giving us the opportunity to save a huge number of lives.