LOGIC MODELS: A ROADMAP FOR PROGRAM PLANNING, IMPLEMENTATION AND EVALUATION

1. LOGIC MODELS: A ROADMAP FOR PROGRAM PLANNING, IMPLEMENTATION AND EVALUATION Dean Schillinger MD Professor of Medicine Director, UCSF Center for Vulnerable Populations @ SFGH Chief, CA Diabetes Program, CA Dept Public Health

2. Program and Program Theory A program is any organized activity or set of public health activities intended to achieve an outcome Program theory: • a set of assumptions about how a program will produce desired results • explains why a program will do what it does • provides the rationale for expecting that doing things that way will achieve the desired results • for an outcome and process / performance monitoring evaluation, should be supported by best practices, proven interventions

3. Logic Models and Program Description Logic Models Graphic depictions of the relationship between your program’s activities and its intended effects 3

4. Program/Intervention Logic Model • Defined as a picture of how your organization does its work – including the theory and assumptions underlying the program • Links outcomes (short and long-term) with program activities/processes and the theoretical assumptions/principles of the program

5. Program/Intervention Logic Model 3. Systematic and visual way to present and share your understanding of the - relationships among the resources you have to operate your program, • activities you plan, and • the changes or results you hope to achieve

6. Value of Logic Model Much of the value in a logic model is in the process of creating, validating, and modifying the model. The clarity of thinking that occurs from building the model is critical to the overall success of the program.

7. Step 2: Describing the Program:Logic Model Archtype Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions 7

8. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions What the program and its staff actually do Results of activities:Who/what will change? 8

9. Logic Models • May change throughout the life of the program! • Fluid; a “working draft” • Responsive to lessons learned along the way • Reflect ongoing evaluation of the program

10. Useful for all parties involved(Funder, Board, Administration, Staff, Participating organizations, Evaluators, etc.) • Convey purpose of program • Show why its important • Show what will result • Illustrate the actions that will lead to the desired results • Basis for determining whether actions will lead to results! • Serves as common language Enhance the case for investment in your program!

11. Strengthen Community involvement • Created in partnership, logic models give all parties a clear roadmap • Helps to build community capacity and strengthen community voice • Helps all parties stay on course or intentionally decide to go off-course • Visual nature communicates well with diverseaudiences

12. CREATING AND USING A LOGIC MODEL

13. Two Kinds of Logic Models • Basic, or Program Implementation Model • Theory of Change Logic Model

14. Basic Logic Model 1. Resources/ Inputs 2. Activities 3. Outputs 4. Outcomes 5. Impact Planned Work Intended Results *From W.K. Kellogg Foundation Logic Model Development Guide

15. Elements of aBasic Logic Model • Inputs (resources) • Activities (interventions) • Outputs (units of service, persons served) • Outcomes (short, intermediate and long-term) • Some models also include other elements such as problem statement, assumptions, environment and program target, Impacts

16. A Logic Model Framework Program P R O B L E M INPUTS OUTPUTS OUTCOMES ASSUMPTIONS 1. 2. 3. ENVIRONMENT (external factors that influence outcomes)

17. Components of a Simple Logic Model, follow if statements INPUTS OUTPUTS OUTCOMES • Resources Short term Inter- mediate Activities Participation Long term What is done Who is reached What is invested With what results

18. Basic Logic Model Context Implementation Outcomes FormativeEvaluation SummativeEvaluation (Improve) (Prove)

19. INPUTS/Resources STAFF MONEY LOCATION VOLUNTEERS PARTNERS EQUIPMENT TECHNOLOGY TRUST

20. OUTPUTS What program does Who it reaches ACTIVITIES Treatment Classes Counseling Skill development Case management Curriculum design Trainings Conferences PARTICIPATION Participants Providers Users

21. PROGRAM OUTCOMES What results for individuals, agencies, communities..… SHORT Learning Access Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation INTERMEDIATE Action Behavior Practice Decisions Policies Systems change LONG-TERM Conditions Mortality Morbidity Quality of Life Environmental

22. Assumptions and Environmental Factors • Assumptions Proven theories (research literature/best practices Promising program results Experience Expected conditions • Environmental Factors What besides program interventions can account for program results

23. Logical Linkages: ExampleSeries of If Then RelationshipsProblem: Child Abuse IFTHENIFTHENIFTHENIFTHEN Program invests time & money Design parenting curriculum Parents increase knowledge Parenting improved Decrease rates of child abuse INPUTS OUTPUTS OUTCOMES

24. Problem: Child Abuse Example INPUTS OUTPUTS OUTCOMES Parents increase knowledge of child develop. Design parent ed. curriculum Staff Parents use improved parenting skills Reduced rates of child abuse & neglect Targeted parents attend Money Provide 6 training sessions Parents learn new ways to discipline Partners

25. Questions: Data: Were sessions delivered? # of sessions delivered # of target parents attending session? demographics of parents Increase in knowledge/skill? Pre-post session survey Behavior change? follow-up interview/obs Decrease in rates? from program records Where Does Evaluation Fit? INPUTS OUTPUTS OUTCOMES Parentsincrease knowledge child develop. Design parent ed. curriculum Staff Parents use improved parenting skills Reduced rates of child abuse & neglect Targeted parents attend Money Provide 6 training sessions Parents learn new ways to discipline Partners EVALUATION: What do you want to know? What data do you need?

26. Problem: Program: INPUTS OUTPUTS OUTCOMES Resources Activities Participation Short Intermediate Long-Term PROGRAM ASSUMPTIONS ENVIRONMENTAL FACTORS Logic Model: WORKSHEET

27. CA Diabetes Program Logic Model

28. Assessing Program Logic:Basic Questions How well conceptualized is the program? Does it address the causes/risks of the problem it is addressing? Is it specific enough? well-defined? Are outcomes related to the program’s interventions? Is it reasonable? feasible? Is the program logic valid?

29. Review of Benefits ofUsing a Logic Model: • Helps clarify program rationale and expectations • Identifies gaps in logic and uncertain assumptions • Summarizes complex program to communicate with stakeholders/others • Helps identify appropriate evaluation questions • Builds understanding and consensus (shared approach) • Provides an Evaluation Framework

30. The Value of Logic Models 30 30

31. Helpful Resources NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: http://www.cdc.gov/eval/whatsnew.htm Logic Model Sites Innovation Network: http://www.innonet.org/ Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/ University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/ CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/resources.htm#4 CDC/STD: http://www.cdc.gov/std/program/progeval/TOC-PGprogeval.htm Texts Kellogg Foundation Logic Model Development Guide: www.wkkf.org W.K. Kellogg Foundation Evaluation Resources: http://www.wkkf.org/programming/overview.aspx?CID=281 Rogers et al. Program Theory in Evaluation. New Directions Series: Jossey-Bass, Fall 2000 Chen, H. Theory-Driven Evaluations. Sage. 1990 31

32. Exercises? • Need volunteer(s) and their associated programs

33. Homework • Using the template from class (available from the Kellogg materials provided in the readings), complete the logic model that reflects the content you proposed in class for your area of focus. For those whose project was developed as a class exercise, focus on the theory-based logic model.