Charity Update I: Development Media International

This is the first of a series of blog posts on charities we’re re-evaluating this year.

Development Media International (DMI) is an organization we’ve had our eye on for some time. Back in 2013 we interviewed their Head of Public Engagement, Will Snell, and in May last year we discussed them in greater detail.

To recap what we already know, DMI is an organization focussed on a health intervention that is potentially very effective, but difficult to assess: mass-media campaigns. Their main technique is short, targeted radio spots of 60 seconds or less. These provide small pieces of health information in a memorable form, and the hope is that if people actually respond with behavioural change, then we may see a substantial drop in child mortality.

The big question is how much of an effect DMI’s efforts actually have. Their starting point is a statistical model which they developed in conjunction with the London School of Hygiene and Tropical Medicine. This estimates a reduction in child mortality of between 16% and 24%. Such a high effectiveness is consistent with previous analyses, such as the WHO-CHOICE report that ranked media campaigns very highly as a means for combatting HIV/AIDS. That’s a good starting point, but what really excites us is that they’ve been following this up with a large-scale randomized-controlled trial across Burkina Faso (see the previous interview for more details about the RCT).

What’s new

The RCT is a long-running procedure: it started in 2012, and the full, end-line results are not expected until the end of this year. But DMI have recently released the midline survey results, which give us some idea of what to expect.

The midline results come in the form of survey results measuring various self-reported metrics across the treatment and control groups. Overall, they look promising, showing a consistent improvement in the treatment group. Even after discounting their estimates significantly, GiveWell give their current estimate as $5,236 to save a child life. That’s within a factor of two of some of our other recommended charities (GiveWell estimate that AMF, one of our top charities, saves a child’s life for $3,340), and DMI’s own estimate is much lower.

However, we do have some concerns about the interpretation of these results. GiveWell provides a comprehensive list, but we’d like to highlight just three points here:

1. The results come from self-reported survey data. This means they are always going to be less reliable than assessment of objective measures, such as mortality data.
2. Lack of blinding. Both the participants and the surveyors may have been aware of which group the participant was in - the survey itself contained questions on whether they had heard the radio spots.
3. Asymmetries between the control and treatment groups. There were some relevant health differences before the treatment began, and there were other health programmes running simultaneously with the RCT.

The self-reporting and blinding problems will not persist to the endline data - for that, DMI are going to measure mortality statistics. However, they do present problems for the midline data. The lack of blinding is the most worrying: lack of blinding can lead to overstated treatment effects, and its effects are hard to account for in retrospect. However, DMI have confirmed that the questions which specifically relate to their programmes appear in the last section of the questionnaire. That should mean that the interviewees will only be unblinded for the last section at worst, although the interviewers are still unblinded. While not ideal, this is much better than having a full lack of blinding.

The problem with asymmetries between groups will remain even for the endline results. GiveWell highlights three prominent health programmes that might have had some effect on child mortality during the running of the trial. By and large, it seems likely that the effects were present in both groups, and the independent evaluators at the LSHTM plan to adjust for such baseline imbalances between the control and intervention arms in their analyses.

Finally, we have some worries about the portability of DMI’s programmes. Unlike many other health interventions, which have a biological pathway of action that we can expect to work consistently across humans, DMI’s radio spots rely on culturally changing behaviour. As such, they may run into communication barriers that are hard to discover - in one country an older female voice might be regarded as authoritative, whereas in another the reverse might be true. However, DMI have extensive experience in developing these programmes, and perform a great deal of formative research and testing for efficacy. If the RCT provides an endorsement of the overall methodology, then we are fairly confident that cultural hurdles can be overcome.

Our conclusions

Overall, we broadly agree with GiveWell that DMI is an excellent organization with plenty of room for funding, and that the midline RCT results are promising. DMI’s approach of spreading simple, but effective, health information is very scalable, and there are a wealth of possible extensions to the programmes (for example, further simple improvements in post-natal care and food preparation). With that in mind, we consider DMI to be a promising charity, and if the endline results from the RCT prove good, we are likely to consider them one of our top charities.

Thanks to DMI for clarifying several points with us directly, and to GiveWell for an excellent baseline report.