The Comparative Method for Assessing Impacts and Cost Effectiveness of Development Assistance Programs

1. The Comparative Method for Assessing Impacts and Cost Effectiveness of Development Assistance Programs Paul Clements Western Michigan University

2. It is possible to substantially enhance the impacts of development assistance with evaluations employing the comparative method. This depends on two, related arguments: • An analytic argument 2. An organizational or structural argument

3. The analytic argument - overview Linear analysis, the method for rigorous impact evaluations (e.g. with randomized controlled trials) is grounded in situational analysis. Situational analysis can be rendered more rigorous through the comparative method with synthesis anchored in cost effectiveness. Applying lessons from evaluations always requires (an act of) synthesis. Weak synthesis in development practice is due about equally to analytic shortcomings and to weak incentives (and the melding of these in organizational cultures). Stronger synthesis (hence more cost effective development interventions) can be achieved through a particular organization of evaluation based on the comparative method.

4. The organizational/structural argument - overview (See “Reducing World Poverty by Improving Evaluation of Development Aid,” by Paul Clements, ThomazChianca and Ryoh Sasaki, American Journal of Evaluation, June 2008, pp.195-214.) Aid evaluation today, e.g. under the DAC criteria or results-based approaches, has uneven quality and inconsistent units and evaluative judgments even for similar projects. It is controlled by donor and implementing agencies with incentives for positive bias, so evaluative conclusions tend to be too narrow (not addressing cost effectiveness) or positively biased, and both of these amount to positive bias. Evaluations are usually framed in individual, project specific terms. Together these factors undermine learning and accountability in the development assistance community.

5. The proposal - overview It is possible to dramatically improve the quality of evaluations, the discourse of the development community, and the impacts of aid interventions, by professionalizing aid evaluation. A professional association (Evaluation International) with guidelines and standards for evaluation based on impacts and cost effectiveness. The association’s evaluations also: • explain impacts in terms of project design and implementation, - address strategic issues in the sector, and • employ the comparative method. Units and evaluative judgments are consistent between evaluations.

6. The Comparative MethodIntroduction Cost effectiveness is the correct analytic anchor for the discourse of development management (we want to maximize impacts given our resources). Linear Analysis Rigorous impact estimates based on the well known formula: (Ben t1 – Ben t0) – (Control t1 – Control t0) Sets of statistical procedures and analytic requirements for valid and precise results. Linear analysis depends on situational or constructive analysis for setting up the design and for applying the results. The comparative method builds on and adds rigor to situational or constructive analysis

7. The Comparative Method:situational/constructive analysis Integrates sense of mechanism, of causal relations, vis a vis an objective Involves our sense of the relative importance of several contributing factors Involves triangulation

8. The Comparative Method:Forms of Reasoning (a) Given similar known complex phenomena a, b, c and a new one, x a > b > c ; where does x fit in this range?

9. The Comparative Method:Forms of Reasoning (b) Consider two microfinance programs in environments with: Program A Program B High and variable inflation Low and steady inflation Scattered, sparse population Dense population Largely subsistence-based Fully monetized society society Low GDP growth High GDP growth

10. The Comparative Method:Forms of Reasoning (c) Two microfinance programs: Program A Program B Weak training Strong training program program Weak sense of Strong sense of mission mission

11. The Comparative Method:Forms of Reasoning (d) Estimating future impacts Evaluation of condition C at time t1 Ct1 – Ct0 = x what is Ct2 – Ct1?

12. The Comparative Method:Forms of Reasoning (e) Routine, every-day forms of reasoning Employed in complex integrated judgments (hence constructive) When grounded in a single value – cost effectiveness – exceed lay person’s normal analytic capacity Can be systematized

13. The Comparative MethodIntegrated To evaluate a primary health care project: Assume 20 evaluations of completed primary health care projects, more and less rigorous, all estimate impacts and cost effectiveness All explain impacts and c. e. in terms of the project’s design and implementation and characteristics of the target population and of the context or environment. • Evaluator locates the project on the multi-dimensional spectrum established by these earlier evaluations • Evaluator investigates hypotheses based on this placement • Analysis of baseline and counterfactual • Sample survey to estimate changes in conditions • Estimates future impacts and costs • Inferences informed by technical literature and prior evaluations as well as survey results, project records, etc.

14. The Comparative MethodHow to Develop and Apply it This can be achieved, rendered rigorous, and generalized through the establishment of a professional association along the lines of associations of accountants and auditors. Proceeding on a sector basis: Consistent units, consistent evaluative judgments given similar evidence.

15. The Comparative MethodSupports Learning and Accountability Given consistent estimates of cost effectiveness • Better portfolio management by donor and implementing agencies • Stronger within- and across-sector learning across the development community • New basis for personnel management • More cost effective resource allocation