60 likes | 231 Views
The complex evaluation framework. Simple projects, complicated programs and complex development interventions. country-led planning and evaluation Non linear Many components or services Often covers whole country Multiple and broad objectives
E N D
Simple projects, complicated programs and complex development interventions • country-led planning and evaluation • Non linear • Many components or services • Often covers whole country • Multiple and broad objectives • May provide budget support with no clear definition of scope or services • multiple donors and agencies • context is critical Large, complex Complex interventions • May include a number of projects and wider scope • Often involves several blueprint approaches • Defined objectives but often broader and less precise and harder to measure • Often not time-bound • Context important • multiple donors and national agencies Complicated programs Simple projects Small, simple • “blue print” producing standardized product • relatively linear • Limited number of services • Time-bound • defined and often small target population • Defined objectives
The Special Challenges of Assessing Outcomes for Complex Programs • Most conventional impact evaluation designs cannot be applied to evaluating complex programs • No clearly defined activities or objectives • General budget and technical support integrated into broader government programs • Multiple activities • Target populations not clearly defined • Time-lines may not be clearly defined
Special challenges continued • Multiple actors • No baseline data • Difficult to define a conventional comparison group
Alternative approaches for defining the counterfactual for complex interventions 1. Theory driven evaluation 3. Quantitative approaches • Qualitative approaches • Mixed method designs • Rating scales • Integrated strategies for strengthening the evaluation designs
STRATEGIES FOR EVALUATING COMPLEX PROGRAMS • Counterfactual designs • Attribution analysis • Contribution analysis • Substitution analysis Estimating impacts The value-added of agency X Theory-based approaches Net increase in resources for a program Mixed method designs • strengthening alternative counterfactuals • “Unpacking complex programs” • Portfolio analysis • Reconstructing baseline data • Creative use of secondary data • Secondary data • Triangulation Qualitative approaches Quantitative approaches Rating scales