THE IMPORTANCE OF EVALUATION

In the 2003 article by Jones and colleagues published in The Lancet, information was assembled on a set of low cost, feasible services that if universally available to poor mothers of the world would be expected to reduce the number of children that died by two thirds – some 6 million each year.  This suggests that an effective effort would be one that worked to make such services widely available in a country.  It would not be feasible to try to design one health program that would bring all these services to all mothers.  Rather the needed efforts might proceed as follows, and let us think in terms of how USAID might proceed.

1. Since in many, perhaps all, poor countries there would be economic, medical, establishments that would be inclined to try to limit the range of efforts they would support, the first thing needed for effective action would be to encourage the creation of a National Health Council (NHC) that would develop plans, with technical assistance of groups like USAID.

2. Let us consider pneumonia, typically the first or second killer of children. The NHC might develop an experimental program in which mothers would be trained to recognize pneumonia – 50 breaths or more per minute.  For this, mothers would need to be provided with a clock with a second hand.  Mothers would have on hand antibiotic powder, know how to mix it with water, have a dropper and know how to administer to their sick child.  There are obvious problems in all of this, with solutions to be found.  Treatment of diarrhea would be another challenge.

3. Such experimental programs would need to be evaluated.  This means initiating the experimental program in a sample of villages drawn from a population of villages and comparing child deaths in this sample with the deaths in a control sample drawn from the same population. 

USAID has not carried out these types of evaluations.  We can work with them to improve their effectiveness.  The NHC, being a national group, can play an important role in getting agreement on such evaluations.  We can see the tragedy of the lack of a correct evaluation design in UNICEF’s $27 million effort in Africa.  Experimental and control groups were selected from different populations, so no conclusions on statistical grounds can be drawn. 

The American Statistical Association, in action initiated by its Human Rights Committee, of which I am a member, is planning to try to work with USAID to help them to be more effective. I have made suggestions for the organization of a session on evaluation, which has been done.  I will be chairing the session.  Evaluation in international health is very weak.  Hopefully the session will get us thinking about how we might play a more active role in this area.   

 --David J. Fitch, PhD, Instituto de Investigaciones Económicas y Sociales, Universidad Rafael Landivar, Guatemala