Logic Models for Evaluation: The Search for Clarity

1. Logic Models for Evaluation:The Search for Clarity Gail Johnson Spring 2008 logic models class.ppt

2. Program Evaluation • Common analytical technique in public administration and non-profit programs • Purpose: to tell what happened during the implementation of a program, tell how well a program is working, and whether it is achieving the desired results. • Two main types: • Formative: focus on process • Summative: focus on outcomes logic models class.ppt

3. Problem • For too long, public agencies (and non-profits) have focused on reporting budgets and activities. • Program managers: get enmeshed in fighting the alligators that they forget that the purpose of the organization is to drain the swamp. • Captured by the demands of the job, lose the big picture logic models class.ppt

4. Emergence of Logic Modeling • Logic modeling for evaluation grew out of two other techniques: • Systems Thinking • Performance Measurement logic models class.ppt

5. Systems Model Black Box Environment Inputs Outputs logic models class.ppt

6. Lessons Learned from Systems Thinking • The importance of the environment—or context • You can have a great program for empowering women through micro-lending, but if a civil war breaks out, it chnages everything. • Complexity: Zoom-in and zoom-out: • Logic models within logic models • Challenge: to find the right focus • Non-linear nature: importance of feedback loops logic models class.ppt

7. Performance Measures • GPRA, GMAP, etc • Move from focus on Process to focus on Outcomes: What changes because of the program? • Need to focus on indicators to measure change • Search for indicators logic models class.ppt

8. Thinking Differently: Working with Logic Models • Models help us visualize a program in context • Models help us identify the relationships between various components: • Models help us think more analytically about cause and effect: • Describes the theory of change guiding the program intervention • Models help us identify key assumptions that can be tested logic models class.ppt

9. Models: Cause and Effect Did the program cause something to happen? Education Employment logic models class.ppt

10. SystemsTheory-based Evaluation • Its focus is to understand the nature of the problem and the relationships between the problem, intervention and expected outcome. • What is the theory? • Why do we believe that training people will result in better evaluations? • Why do we believe that funding micro-businesses will reduce poverty? • If an assessment about the relevance is to be made, the theory has to be examined. logic models class.ppt

11. Theory and Causal Chains • Training in Evaluation High Quality Evaluations • Training in Evaluation High Quality Evaluations  Useful Information • Training in Evaluation High Quality Evaluations  Useful Information  Better Decisions logic models class.ppt

12. Logic Model Impact Inputs Outcomes Outputs Activities logic models class.ppt

13. GoalImpacts) • Long-term, widespread improvement in society • Intermediate effects of outputs on clients Outcomes • Products and services produced Outputs • Tasks personnel undertake to transform inputs to outputs Activities Inputs • Financial, human, and material resources Logic Model: Results Implementation logic models class.ppt

14. Higher income levels; increase access to higher skill jobs Goal(Impacts) • Increased literacy skill; more employment opportunities Outcomes • Number of adults completing literacy courses Outputs Activities • Literacy training courses Inputs • Facilities, trainers, materials Logic Model: Adult Literacy logic models class.ppt

15. Logic Model • Outcomes/Impacts: • The changes to expect to see if your theory is correct and the program has been implemented well • What would be different because of the program? • Observable and Measurable Indicators ofChange • Outcomes: Initial, intermediate, and long-term change • Impact: the big enchilada, long-term, related to program goal • Reduce Poverty logic models class.ppt

16. Logic Models • If we train the chronically unemployed, then their quality of life will be improved, and the overall poverty rate will be reduced. • What is the underlying theory? • What’s the model? Unemployed  Training  Employment  Increased Income Reduced Poverty logic models class.ppt

17. Logic Model: Training Program Inputs Activities Outputs Outcomes Impacts • Training programs • Dress for success coaching • Interview coaching • Resume assistance • Products • Number of graduates per training session • % graduate rate • Benefits changes • Increased skills • % Obtain jobs • % Obtain high paying, quality jobs • Increased self-esteem • Goals • Increased income • Self-sufficiency • Family stability • Reduction in poverty • Resources • money • staff • Supplies • mentors logic models class.ppt

18. Exercise: Women’s Program • Women-owned micro-enterprises • Handout • Put in logical order: • Sequence of Inputs, Activities, Outputs and Outcomes logic models class.ppt

19. Logic Model: The Process • It brings in all the key players: stakeholders, staff, clients/customers, political folks, and researchers • It is a process that helps everyone get clear about what it is they are trying to accomplish • It is a living thing: changes • Handout: Compare Figure 6.4 and figure 6.8 logic models class.ppt

20. Complexity • Challenge: level of detail • Handling multiple theories of change • Logic models within logic models • Zoom-in or Zoom out: trying to get the right focus • Accurate, clear but not overwhelming logic models class.ppt

21. Logic ModelEvaluation • When done right, sets the focus for the evaluation (Figure 6.8) • Focus on activities: • Did they do what they said the were going to do? If not, what barriers were in the way? • Focus on outcomes: • 5 critical outcomes • Do all, some, time-sequence? logic models class.ppt

22. Exercise • Familiar Program • Goals? • Objectives? • Activities? • Underlying theory logic models class.ppt

23. Applying the Program Outcome Model Outcomes Impact Inputs Outputs Activities logic models class.ppt

24. Challenges in Developing Logic Model • Outputs and activities are seen as outcomes • Program managers focus on actions • Visible and more within program managers’ control • Easier to confirm task completion than measure an outcome • Measures are not easy to come by • Not all the can be counted really matters and all that matters can not necessarily be counted logic models class.ppt

25. Logic Model: No Panacea • Can’t fix a situation where there is no real theory of change • Still need a rigorous research design • Other alternatives might work better • Stuckness • No intervention exists in isolation • Larger events may be uncontrollable. logic models class.ppt

26. Summary of Program Outcome Model • Identify components of a program • Identify relationships between the program: expected and actual • Identify what we want to measure • Focuses our attention on outcomes and impacts logic models class.ppt

27. Resources • Joy A. Frechtling, Logic Modeling Methods in Program Evaluation, Wiley and sons, 2007. • Kellogg Foundation: Logic Model Guide http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf • CCD: Logic Model Resources http://www.cdc.gov/eval/resources.htm#logic%20model logic models class.ppt