Interview with GiveWell: giving to learn, research methods, and the added value of AMF

Research Director Robert Wiblin interviewed GiveWell co-founder Elie Hassenfeld on a range of issues suggested by volunteers and members of Giving What We Can. This is the second post, which covers when to donate, how GiveWell's research has changed over time, and exactly what the Against Malaria Foundation are doing that makes them GiveWell's number one charity. An unabridged transcript of the interview is available here and MP3 audio here.

RW: Do you think donors should consider waiting a couple more years in the hope that you or someone else will be able to find a significantly more cost effective option that AMF or your other recommendations?

EH: This is a topic Holden wrote a post on a while back, but I think there's a couple of things to think about. I think that if I had to bet, I'd bet that our recommendations three years from now will be better than our recommendations today. I do think there's a couple of pretty important counter-arguments to that and reasons why I personally give to our current recommendations, rather than just waiting around for our recommendations a few years from now. One is I think that it's helpful to become habituated to giving to charity. I think that there's a non-negligible danger of allowing yourself to hold on to the funds and never quite being ready to pull the trigger. Obviously, there's an easy way to deal with that problem, which is using a mechanism like a donor-advised fund where you actually get the money out the door, but you don't select the organisation to which you're giving.

The other factor that I also think is important is giving to learn. By that I mean giving money and then following up with the organisation to find out what happened. You gave the money as a means for improving your understanding about how the charity world works. And this has been one of best learning mechanisms that we have found at GiveWell. It goes back to our recommendation of VillageReach and all of the work we've done to follow them over the years to understand what went well and what went poorly for that organisation. But the same is true for SCI, AMF, GiveDirectly. Following closely on the ground is the way we learn about effectiveness. If everyone were to decide not to give and to hold out, I think the quality of GiveWell's future research would fall because we wouldn't have that opportunity to direct money and learn from how we directed that money.

RW: Related to that, one of our members sent the question "How many charities do you need to investigate in each field before you get confident that you're getting near the best?" I guess I have a similar question: how your recommendations have changed over the last seven years? Have they been getting better or worse and why, and does that inform your best guess as to whether your recommendations will get better or worse in the future?

EH: Let me answer those in turn. The first question was about the number of charities you investigate in a field before feeling confident that you're getting 'near the best'. I don't really think the set number is the right way to think about it. You want to do enough work in a given area that you start to see patterns in the organisations you're finding and you're not surprised by the results that you find. You're not bumping into new organisations, new approaches, new ideas that are things you haven't investigated at all or can't really know how they would match up to our top charities. For us, in the area of global health and development, over the years we've been in business, we've looked into this area sufficiently that we haven't been surprised for a long time at something that's just popped up on the scene, we generally see the same stories popping up over and over again.

I think we have a good sense of the characteristics of an organisation that would make them promising in our process - that they are largely implementing a program that has strong evidence behind it, and is cost-effective, and that they are willing to be transparent. You do those things, that's great. You don't do that, that's less promising. And we've got to the point where we're not just bumping into organisations that have those characteristics that we haven't heard of before. So that means that at this point, with our global health and development organisations, I'm less optimistic that we're going to find new organisations. I know who the next ten contenders are and we're trying to encourage them to engage us, but we also have a very good sense of the lay of the land.

On the second question - of changes over time - I think that our research has become significantly more intense and in-depth, and I do think the recommendations that are on our site now are better in some aggregate sense than the recommendations we made in the past. We went full time in 2007. In our first full-time period, we received grant applications from organisations and we had very little information. We tried to do these very explicit cost-effectiveness calculations for the organisations that applied, and we recommended Population Services International and Partners in Health based on these very explicit cost-effectiveness calculations. With Partners in Health, there was some of the case that was not explicitly quantifiable. All of these archived reviews are available on our website if people are interested in seeing the 2007 version of GiveWell's research.

Our research back then was limited both by the set of organisations that applied, but also our understanding of what we should be looking for. In 2009, we changed our process, we reached out to new organisations, and that's when we recommended Village Reach and Stop TB Partnership as our top organisations. Both Village Reach and Stop TB Partnership are incredibly outstanding organisations, certainly in terms of the programs that they work on, how transparent they are, the quality of people involved in their activities. But they have weaknesses that they have that our current charities do not have.

In the case of Stop TB, we couldn't get compelling information on room for additional funding. What was really going to happen to Stop TB's programs because of additional funding from GiveWell directed donors? At the time of our review, we thought they were going to provide more basic treatments for TB. Back to that learning idea, we were going to follow that up, and we learned later that the funds were not needed for first-line TB drugs - the basic TB treatment. We wrote in the latest review of Stop TB Partnership that we don't have sufficient understanding of room for more funding to recommend this organisation.

Today, our recommendations are ones where the room for more funding - what impact an additional dollar donated to the organisation has - is a major part of what we're focussed on. In the case of Village Reach, we knew that they were taking a riskier approach to development than we had been hoping for when we started looking for top charities. The thing that they didn't have was that really iron-clad evidence-based case that their programs made any difference. I think they had a lot of evidence that it did, but there are also questions that were not answered by the types of data they were able to present, more post-hoc assessments of what occurred, and using some qualitative assessments of the likely causes of changes driven by their programs. AMF, Givedirectly and SCI are all organisations whose programs are informed by multiple Randomized Controlled Trials. This is a standard of evidence that is significantly higher than the standard we held Village Reach's programs to.

In the very big picture, the way I would describe the change in our recommendations over time is that we've become more sensitive to the issues that are most important to investigate deeply and we've significantly increased the depth of our research. I look at GiveWell today, and I think what we've done for global health charities is outstanding, but I also recognise that there are certain types of activities that we have not looked into solely because we haven't had the time yet to get to them. Activities like funding scientific research, or potentially trying to influence government action. These are things that large-scale philanthropists are very involved in.

It seems to me intuitive that they might be significantly more impactful than organisations that provide direct services, and therefore we must look into that. We might not find anything. But my instinct is we'll find things that at least for some donors are significantly more appealing than the organisations that we recommend today. I think that will largely come by broadening the scope of the research we've done to remove the criterion of significant evidence of effectiveness.

RW: You mentioned earlier that it seems like you can both give now to your best charities and support incredibly valuable research into what works. So what kind of research are the charities you recommend - AMF, SCI and Givedirectly - doing? What's a specific project that they're conducting which excites you and is going to produce useful information for GiveWell and others?

EH: Well, AMF is on the ground giving out bednets and then they're tracking what's happening to malaria rates in communities in which they're working. They're tracking net usage and posting it all on their website. They're also intending - though I don't have the specific details into when and how this will play out - to track any evidence of mosquito resistance to insecticide-treated mosquito nets. Obviously they're tracking malaria rates, so that's helpful.

These are the things that if I had never come into contact with the malaria community before, I would have thought these must be tracked - it's obvious - you track malaria rates, track usage, track insecticide resistance, but this data is not readily available. If you look at the page that we published on insecticide-resistance and malaria nets last year, you'll see that there is very limited information out there about this.

AMF is on the ground doing a few things: first, obviously, the malaria nets save lives, but because they are transparently sharing everything that happens on their program, they are helping us understand the challenges of direct provision of nets, they are also collecting and sharing information that I think will help the malaria community more broadly. In the event that AMF gives out nets and malaria rates don't fall, that will certainly be an instance where one will be able to learn a great deal about the circumstances in which nets may not work. So far, malaria rates have fallen significantly since the nets were distributed.

The same is true for GiveDirectly. They are collecting and sharing information about how people use the funds that they receive. GiveDirectly itself is involved with an RCT on the impact of cash transfers and studying cash transfer mechanisms, for example giving to a subset of a village versus giving to every single person in a village.

--

Image credit: CC-BY 2.0 Radio Okapi