Note: This article hasn't been updated recently. We're keeping it up for the historic record of the debate, and because people still seem to refer to the arguments. Giving What We Can no longer actively conducts charity evaluation. We recommend you read GiveWell's latest research on GiveDirectly for the most recent evaluations.
GiveDirectly is a charity founded in 2008 that provides unconditional cash transfers to households in extreme poverty. GiveDirectly is currently among GiveWell's top recommended charities, but doesn't feature on our own list of recommended charities. In this post, I'll explain why we haven't changed our mind about GiveDirectly since we last addressed the issue in late 2012. In particular, I'll discuss the results of a recent randomized control trial (RCT) from November 2013, which some have taken to answer skepticism about unconditional cash transfers[1]. While generally positive, the results of this RCT also provide evidence that transfers are less cost-effective than was indicated by estimates made by GiveWell when we last commented on their recommendation. It goes without saying that we have great admiration for GiveWell's work, and much of our own research is heavily indebted to theirs[2]. However, on this point we continue to respectfully disagree.
What is GiveDirectly?
At present, GiveDirectly operates in Kenya and Uganda. Transfers are targeted at the very poorest, and GiveDirectly employs a detailed selection procedure to identify the worst-off households. Within villages, these are selected based on homes of mud and thatch that lack metal roofs[3]. Marginal donations to GiveDirectly go toward cash grants, as operating costs are independently funded by large donors. Transfer costs absorb approx. 10% of donations, meaning that roughly 90% of the amount given ultimately reaches recipients[4]. Cash transfers are a well-studied intervention-type, beginning with highly encouraging results obtained in a randomized trial of the Mexican Progresa programme[5]. Within the field of cash transfers, GiveDirectly's programme has two distinctive features.
Firstly, transfers are unconditional: recipients are not obligated to spend their transfer in any way, and receipt of further instalments is not dependent on meeting goals associated with, say, children's education or health (as in the Progresa programme). Secondly, relative to other cash transfer programmes, the amounts transferred are large. Households receive approx. $1,000 (in instalments) over the course of a year[6]. By way of comparison, this represents approx. 55% of median household income[7]. The same amount of money could be used to supply long-lasting insecticide-treated nets to more than 160 people[8] and provide deworming treatment to more than 1,000 children[9]. Although cash transfers are well-studied, there is relatively little direct evidence on the impact of large, unconditional transfers of this kind. However, there does exist a recent, well-designed, pre-registered RCT that examines short-term impacts from GiveDirectly's own programme. Results from this trial were reported in a policy brief by Johannes Haushofer and Jeremy Shapiro released in November of 2013[10].
GiveDirectly as a recommended charity
GiveDirectly appeared on GiveWell's list of top recommended charities in the winter of 2012, ranked above Schistosomiasis Control Initiative (SCI) and below Against Malaria Foundation (AMF). At the time, GWWC Vice President William MacAskill offered a broad range of considerations explaining our skepticism about GiveWell's recommendation and our continuing decision to preferentially recommend health charities[11]. Will noted a number of general considerations that should lead us to assign low prior confidence to the idea that cash transfers are more cost-effective than the best health programmes. In addition, he pointed to the paucity of evidence on the impact of large, unconditional transfers and expressed surprise at what he regarded as a lack of concern for cost-effectiveness underlying GiveWell's recommendation.
At the end of 2013, GiveDirectly maintained its status among GiveWell's top-recommended charities. GiveWell no longer explicitly ranks its charities, though senior staff members' personal donation-decisions suggest they regard GiveDirectly as the best option for giving[12]. GWWC has maintained its position on GiveDirectly. Maintaining our position requires some explanation in light of the results reported by Haushofer and Shapiro, which go some way to addressing Will's earlier concern about paucity of evidence. Ultimately, disagreement between GiveWell and GWWC on GiveDirectly reflects more fundamental methodological divisions. GWWC places greater weight on explicit estimates of charity cost-effectiveness, which GiveWell regards with more skepticism[13]. Consequently, GiveWell's recommendations lend greater weight to assessments of organizational virtues, such as transparency, self-monitoring and evaluation, etc. Our own recommendations weight organizational virtues to the extent that they provide reason to believe that an organization can reliably deliver an intervention whose cost-effectiveness can be pre-established with reasonable certainty. GiveWell's own cost-effectiveness estimates suggest that GiveDirectly's wealth transfers are less effective than deworming[14]. What we believe about GiveDirectly depends, ultimately, on the weight we give to cost-effectiveness estimates of this kind. Our recommendations lend them more weight than GiveWell's. There is thus a deeper controversy, which we can't hope to resolve here. Instead, I'll confine my discussion to explaining why the results reported by Haushofer and Shapiro haven't changed our minds.
A randomized control trial of wealth transfers
Haushofer and Shapiro conducted a randomized control trial in the Rarieda district of western Kenya in 2011-2012. Whereas our information about charity cost-effectiveness typically relies on evidence regarding the cost-effectiveness of the type of programme being carried out, Haushofer and Shapiro directly assess GiveDirectly's own work, albeit with one qualification. GiveDirectly's transfers are typically worth approx. $1,000. However, only about a quarter of households received transfers of this size, with the remainder receiving approx. $300. The aim here was to assess the impact of transfer size on key variables (described below). However, the overall results of the study are often reported in a manner that averages across both groups and doesn't easily allow us to look within treatment conditions[15]. Haushofer and Shapiro looked at 8 key variables: total assets; total consumption; food security; agricultural and business income; health; education; psychological well-being; and female empowerment. Overall, the results were encouraging, with recipients of larger transfers showing greater improvements. However, some key variables showed no significant positive impact.
Recipients chose toinvest a significant fraction of their transfer, in livestock, furniture, and home improvements. The total value of their asset holdings increased by USD PPP 279 on average, representing a 58% increase over the control group. Relative to controls, recipients of transfers were 23 percentage points more likely to own an iron roof (16% vs 39%); Haushofer and Shapiro estimate a 7-14% return on investment (ROI) for iron roof purchases, as this obviates the need to make periodic, costly repairs of thatched roofs.
Cash transfers increased all categories of consumption, except for temptation goods, such as alcohol and tobacco, with a 19% increase in food consumption relative to controls. Food security increased significantly in transfer recipients, reducing the likelihood that respondents had gone to bed hungry in the previous week by 7 percentage points and the number of days children had gone without food by 42%.
Revenues from agriculture and business increased, though there was no significant increase in profits. Transfers did not significantly increase indices of health and education. Transfer recipients reported subjective well-being greater than that of controls by 1/5th of a standard deviation, though measures of the stress hormone cortisol were not significantly lower than in controls. Female empowerment increased significantly, as measured by attitudes toward women and reports of spousal abuse.
Some variables show 'spill-over' effects, estimated by comparing non-recipient households within villages where some households received transfers with households in villages where no households received transfers. Psychological well-being and female empowerment showed positive spill-over effects. Positive spill-over effects on psychological well-being are especially welcome in this context, as one worry regarding wealth transfers is that they might create resentment among non-recipients. There was some evidence for a negative spill-over effect on consumption, but this was only statistically significantly at a 10% level. Haushofer and Shapiro found that transfers had no impact on the general village-level economy.
The study comes with some limitations. With the exception of cortisol levels, all data is self-reported. This may raise the worry that positive results are driven by recipients' desire to report encouraging results that please experimenters - especially given that measures of cortisol did not accord with self-reported gains in well-being. We're not especially inclined to discount the results on this basis, however, because self-reported results also show no significant impact for a number of key variables, including business profits, health, and education. As Haushofer has noted, the mixed nature of the results provides evidence for internal validity[16]. The more significant limitation is the inability of this study to assess long-term impacts. It is far from implausible that significant positive impacts on business profits, health, and education emerge over longer time-scales.
Should this change our view?
We have never seriously doubted that cash transfers could make improvements in the lives of poor people in the developing world. The only question concerns whether the size of these benefits are comparable to those attained by our recommended charities. In this respect, GiveWell have done a fantastic job in describing models that allow us to compare the cost-effectiveness of deworming and wealth transfers. One such model was described in late 2012[17], and the most recent came out at the end of 2013[18].
The key thing we'll note here is that the estimate produced in 2012 relied on assumptions about cash transfers that appear to have been too optimistic in light of results reported by Haushofer and Shapiro. Therefore, we don't regard these results as providing sufficient reason for us to revise the position we had at that time. Updated in light of other evidence from 2013 indicating lower effectiveness of deworming[19], GiveWell's own estimates continue to indicate that deworming is more effective than cash transfers by roughly the same factor.
Deworming and wealth transfers can be compared by thinking of deworming as an investment that pays off in terms of greater future earnings; the present value of these earnings can be compared with that of investments made by recipients of cash-transfers, primarily in terms of purchases of metal roofs. The case for deworming as a source of greater income in adulthood rests principally on two studies. Based on a retrospective survey from hookworm eradication in the American South, Hoyt Bleakley (2007)[20] predicted that deworming during childhood should raise incomes by over 20% in adulthood in Sub-Saharan Africa; Sarah Baird and colleagues (2011)[21] subsequently obtained evidence from deworming programmes in Kenya that confirm this prediction[22].
Comparing deworming and wealth transfers in this way depends on a number of key assumptions such as the discount rate, the external validity of the results reported by Baird et al., the return on investment for cash transfers, etc. for which different staff members have offered different estimates. The estimates made by GiveWell in late 2012 turned out to be optimistic in two respects.
The first concerns the return-on-investment, though here only some estimates may be thought too high. ROI was originally estimated at 10%, 25%, 5%, or 54%, depending on the person surveyed. As noted, Haushofer and Shapiro estimated a 7-14% ROI for investment in roofs. Business investments showed no significant evidence of increasing profits and thus provide no evidence of additional returns on investment. However, the latter point may simply reflect the short-term nature of the study.
The more significant overestimate concerned the proportion of transfers invested, estimated in 2012 at 75%. Haushofer and Shapiro report that recipients invested $PPP 278 on average, with recipients of larger transfers investing more than recipients of modest transfers by $PPP 253. This suggests that recipients within each group invested approx. 30% of the amount received. Of the $PPP 278 invested on average, $PPP 85 went to livestock, $PPP 53 on durable goods such as furniture and agriculture appliances, and $PPP 10.22 was saved. This may suggest that less than half of invested funds went into purchasing iron roofs.
Conclusion
Although the results obtained by Haushofer and Shapiro may be thought to significantly help the case for large, unconditional cash transfers - and certainly address earlier concerns about paucity of evidence - they are less positive in other respects, particularly when compared with estimates available when we last addressed the value of cash transfers. Ultimately, we don't feel that these results have tipped the balance in favour of GiveDirectly.
Sources:
[1] See the discussion of these results in The Economist: http://www.economist.com/news/international/21588385-giving-money-directly-poor-people-works-surprisingly-well-it-cannot-deal [2] We'd also like to thank GiveWell for comments on a previous draft of this post, which helped to clarify a number of important points for us. [3] http://www.givedirectly.org/howitworks.php# [4] http://www.givedirectly.org/financials.php [5] Banerjee & Duflo Poor Economics, pp. 78-81 [6] http://www.givedirectly.org/faqs.php [7] For median household income figures see: http://www.gallup.com/poll/166211/worldwide-median-household-income-000.aspx#2 [8] Assuming a cost of approx. $6 per net distributed. http://www.givewell.org/international/top-charities/AMF#Costperlifesaved [9] Assuming a cost of less than $1 per child dewormed. http://www.givewell.org/international/top-charities/schistosomiasis-control-initiative [10] http://web.mit.edu/joha/www/publications/Haushofer_Shapiro_Policy_Brief_UCT_2013.10.22.pdf Shapiro co-founded GiveDirectly. [11] https://www.givingwhatwecan.org/blog/2012-11-30/givewell%E2%80%99s-recommendation-of-givedirectly [12 ]http://blog.givewell.org/2013/12/12/staff-members-personal-donations/ [13] See http://blog.givewell.org/2011/08/18/why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/ [14] http://www.givewell.org/international/top-charities/give-directly#Whatdoyougetforyourdollar [15] Overall, households in the treatment group were randomized according to three conditions: the gender of the person receiving the transfer on behalf of the household; the size of the transfer ($300 vs $1,100); and, finally, for smaller transfers, whether these were made as a lump-sum or in monthly instalments.
[16] Here: http://blogs.worldbank.org/impactevaluations/some-thoughts-give-directly-impact-evaluation
[17] http://blog.givewell.org/2012/12/19/cost-effectiveness-of-nets-vs-deworming-vs-cash-transfers/ [18] http://www.givewell.org/international/top-charities/give-directly#Whatdoyougetforyourdollar [19] Sadly, further discussion of this issue is beyond the scope of this post. We hope to further review the status of deworming charities within the recent future. [20] http://qje.oxfordjournals.org/content/122/1/73.short [21] http://www.povertyactionlab.org/publication/worms-work-long-run-impacts-child-health-gains [22] There is also evidence that freedom from childhood malaria considerably increases adult earnings: see references in Banerjee and Duflo, p.45