Beyond logical frameworks to program impact pathways

1. Beyond logical frameworks to program impact pathways CIIFAD M&E Workshop 5 November 2011  135 Emerson Hall Sunny S. Kim, MPH PhD candidate Division of Nutritional Sciences

2. Role of research in evaluation • Evaluation usually focuses on an internal situation, such as collecting data about specific programs, with no intent to generalize the results to other settings and situations. Evaluation determines the merit, worth, or value of things.  • Evaluation asks what should be, which makes reference to the standards and criteria of evaluations. • Program-related research usually conducted with to the intent to generalize the findings from a sample to a larger population. • Research asks why, which indicates a theoretical basis.

3. Basic steps of evaluation • Identifying program goals and objectives • Mapping the program • Selecting an evaluation design • Sample size, control group, sampling frame • Selecting indicators • Determining data collection strategy • Developing data collection instruments • Analyzing the data • Communicating the results

4. Program theory vs. logic model • Program theory explains why a program is expected to work, and a logic model illustrates a program theory. • Program logic model is a picture of how your organization does its work… it links the theoretical assumptions of the program and program activities/ processes and outcomes.

5. Translating program theory… Program theory has also been referred to as: • program logic or intervention logic • theory-based or theory-driven evaluation • theory of change • theory-of-action • impact pathway analysis all ways of developing a causal modal, linking program inputs and activities to intended or observed outcomes

6. Translating program theory… • Program theories are often captured in a series of “if-then” statements. • And between the “if” and “then,” there should be “how” and “why”, i.e. evidence or well-established connections (established theory, previous results, published research, feedback from participants).

7. IF… THEN…

8. IF… THEN… + CONTEXT

9. … into logic models • a commonly-used tool for illustrating program theory. • a picture of describing the sequence of activities to bring about change and how these activities are linked to the results the program is expected to achieve. • often presented in a flow chart that illustrates the linkages between program components and outcomes:

10. Program Logic Model Template

11. … into logic models • “Logic models come in as many sizes and shapes as the programs they represent.” [W.F. Kellogg Foundation, 2001, p.7] • No best logic model, only the model that fits your program best and provides the information you need in the format that is most helpful.

12. Why use a logic model? • Design and planning  helps to explain and illustrate program concepts and approach for key stakeholders (advocate and train); finds “gaps” in the theory or logic of a program and work to resolve them . • Implementation and management  focuses attention on the most important connections between actions and results for monitoring, improved programming, and reporting. • Evaluation, communication, and marketing  presents progress toward goals; provides a way to involve and engage stakeholders in the design, processes, and use of evaluation.

13. Why use a logic model? • Donor requirements: Log frame (OECD/DAC, USAID, CIDA…) Results framework (USAID…) Goal: Health status improved Strategic Objective: Use of key services and behaviors improved Intermediate Result 3: Demand for key services and behaviors increased Intermediate Result 1: Access and availability of services and supplies increased Intermediate Result 2: Quality of services improved Intermediate Result 4: Social and policy environment enabled

14. Logic model example

15. COMPONENTS INTERMEDIATE OUTCOMES FINAL OUTCOMES Maternal and child health and nutrition Improved health and nutrition knowledge and practices Improved health and nutritional status of children <3 years Strategies: Information Education Social communication Basic sanitation and sanitary education Increased access to health services Improved health and nutritional status of pregnant women Strengthening of civil society Improved basic sanitation Small-scale economic activities Improved development plans of CBOs in health and sanitation Intervention model of Programa de Nutrición Infantil (PNI)

16. GOAL: Reduced rates of child malnutrition Strategic Objective: Improved health and nutritional status of children <3 years and pregnant women Intermediate Result 1: Improved nutrition, health knowledge and practices of those attended in the target communities Intermediate Result 3: Improved basic sanitation in the target communities Intermediate Result 4: Community-based organizations applied their improved knowledge on health and sanitation in the formulation of their development plans Intermediate Result 2: Increased access to health services of the target population Results framework of Programa de Nutrición Infantil (PNI)

17. From boxes to arrows… • While “filling in” the logic models with the program components , we must not neglect the mechanisms and linkages… don’t forget about your program theory or impact pathways.

18. Description of mothers workshops “Thetheme/issueisthe training thatyougivethroughtheparticipatorymethdologytothemothers. Ifyoujustgivethetheory, itisthetheme. Theworkshopisthepractice. So ifthatday, forexample, themothersdidn’tbringfoodto prepare theirworkshop, thenthe [health] promoteronlygivesthetheory. Theydiscuss, socialize, and receiveonlythetheories in thesmallgroups… Thentheworkshopreturnsanotherdaytolearnaboutdoing. Forexample, thehealththemes, theymaketheirworkshop do dramatizations. Themothers do theirdramatizations, ittellsthepriority, whathappens and everything, and so they are goingtolearn.” [Regional level, ex-ADRA adviser]

19. Household food availability and access Children consume adequate and appropriate foods CHAs raise topic of child feeding (i.e., BF and CF; balanced diets; feeding sick children) and nutrition for pregnant women Mothers understand nutritional needs of children and pregnant women and know feeding and cooking practices Mothers recall nutritional needs and feeding and cooking practices Mothers implement feeding practices and recipes CHAs and mothers demonstrate recipes CHAs and mothers discuss ideas and experiences Pregnant women consume adequate and appropriate foods Nutritional status of pregnant women improved Husbands permit/ support women to participate Use of local language and readily accessible foods Repeated reminders or internal/external stimulus (e.g. home visits, concern for child’s future) Quality health service access and availability Pregnant women receive appropriate care and health attention in a timely manner CHAs raise topic of health and care for children (i.e., diarrhea and infections; respiratory illness; early stimulation) and reproductive health and prenatal care Mothers understand child health, reproductive health and prenatal care issues and know appropriate practices CHAs convene pregnant women and mothers of children <3 years to participate in workshops Mothers recall child health, reproductive health and prenatal care issues and appropriate practices Mothers implement care practices and seek health services CHAs and mothers discuss ideas and experiences CHAs and mothers perform dramatizations Children receive appropriate care and health attention in a timely manner Child nutrition and growth improved Household access to clean water Mothers understand issues of hygiene and sanitation and know appropriate practices Children exposed to safe and clean home environments and less exposure to contamination Women see incentive to participate (e.g. food baskets or small economic activities) Mothers recall information on hygiene and sanitation CHAs and mothers discuss ideas and experiences Mothers implement hygiene and sanitation practices CHA raise topic of hygiene and sanitation CHAs and mothers demonstrate practices Mothers recall information on self esteem and violence Mothers implement practices related to self esteem and violence Children exposed to safe and caring home environments CHAs raise topic of self esteem and family violence CHAs and mothers perform dramatizations CHAs and mothers discuss ideas and experiences Mothers understand issues of self esteem and violence and know how to address them Influencing factors –facilitators Husbands are sensitized about domestic violence Impact pathway of mothers workshops ?

23. Main references • Kellogg Foundation. (2004). Logic model development guide: Using logic models to bring together planning, evaluation, and action. Battle Creek, MI: W.K. Kellogg Foundation. • Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (Eds), Evaluation models, 209-232. Boston, MA: Kluwer Academic Publishers. • Rogers, P.J. (2008). Using program theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1): 29-48. • Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: teaching and training guide. Madison, WI: University of Wisconsin-Extension.