Logic Models and Organizational Strategy and Evaluation

1. Logic Models and Organizational Strategy and Evaluation By: Thomas J. Chapel, MA, MBA Chief Evaluation Officer CDC/NCCDPHP Tchapel@cdc.gov 404-488-6467

2. Disclaimer… The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the Centers for Disease Control and Prevention.

3. Objectives • Why bother with logic models • Added-value of each component of logic model • Best use of logic models in framing planning and evaluation questions

4. Take-Home Mantras It’s not about the model, it’s about “describing your program” Models make the program logic clear, not true Match the detail/format to the need Simpler is generally better 4

5. Presenting Problem Complex programs operating in complex environments Accountability focused on long-term/distal outcomes Joint efforts of many partners necessary to reach distal outcomes Fragmentation of program planning, performance measurement, and evaluation 5

6. Integrating Processes to Achieve Continuous Quality Improvement • Continuous Quality Improvement (CQI) cycle • Planning—What actions will best reach our goals and objectives. • Performance measurement— How are we doing? • Evaluation—Why are we doing well or poorly? What do we do? Why are we doing well or poorly? How do we do it? How are we doing?

7. You Don’t Ever Need a Logic Model, BUT, You Always Need a Program Description Don’t jump into planning or evaluation without clarity on: • The big “need” your program is to address • The key target group(s) who need to take action • The kinds of actions they need to take (your intended outcomesor objectives) • Activitiesneeded to meet those outcomes • “Causal” relationshipsbetween activities and outcomes

8. Logic Models and Program Description • Logic Models Graphic depictions of the relationship between your program’s activities and its intended effects

9. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions Step 2: Describing the Program:Complete Logic Model

10. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions What the program and its staff actually do Results of activities:Who/what will change?

11. Finding Activities and Outcomes Actions/ Tactics Plan Objectives Goals ST or MT Outcomes LT Outcomes or Impacts Activities Eval Outcome Measures Impact Measures Key Performance Indicators Success Factors Process Measures Progress Measures Impl. Measures PM

12. Lead Poisoning: The Problem • Lead poisoning is a widespread environmental hazard facing young children, especially in older inner-city areas. Exposure lead has been linked to cognitive disruption and behavioral disorders, especially when exposure occurs early in life. The main sources of lead poisoning in children are paint and dust in older homes with lead-based paint. Lead poisoning effects can be ameliorated through medical interventions. But, ultimately, the source of lead in the environment must be contained or eliminated through renovation or removal of the lead-based paint by professionals. Short of that, families can reduce the bad effects on their children through intensive housekeeping practices and selected nutritional interventions…

13. Lead Poisoning: The Program… County X, with a suspected high number of lead-poisoned children, has received money from CDC to support a Childhood Lead Poisoning Prevention Program. The county’s program aims to do outreach and identify children to screen, identify those whose initial and confirmatory screening results show elevated blood lead levels (EBLL) at actionable levels, assess the environments of the EBLL children for sources of lead, and case manage both their medical treatment and the correction of their environment. The program staff will also train families in selected housekeeping and nutritional practices that have been shown to reduce lead levels in homes and lead burden in children. While as a grantee the county can work to assure medical treatment and reduction of lead in the home environment, the grant cannot directly pay for medical treatment or for renovation/remediation of homes. 13

14. Listing Activities and OutcomesLead Poisoning Activities Outreach Screening Case management Referral for medical tx Identification of kids with elevated lead (EBLL) Environmental assessment Referral for env clean-up Family training Effects/Outcomes Lead source identified Families adopt in-home techniques Providers treats EBLL kids Housing Authority eliminates lead source EBLL reduced Developmental “slide” stopped Q of L improved 14

15. Global Logic Model: Childhood Lead Poisoning Program Early Activities Later Activities Early Outcomes Later Outcomes Then…. EBLL kids get medical treatment Family performs in-home techniques Lead source identified Environment gets cleaned up Lead source removed And then… EBLL reduced Develop’l slide stopped Quality of life improves And we do… Case mgmt of EBLL kids Refer EBLL kids for medical treatment Train family in in-home techniques Assess environment of EBLL child Refer environment for clean-up If we do… Outreach Screening ID of elevated kids 15

16. Lead Poisoning: “Causal” Roadmap Activities Outcomes Outreach Do Environment Assessment ID Source and Refer for clean-up Lead Source Removed Train Families Screening ReducingEBLLs ImprovedDevelopmentand Intelligence Family performs in-home techniques Refer for Medical Treatment MedicalManagement ID kids with EBLL MoreProductiveand/or QualityLives Case Management

17. Elaborating the Simple Logic Model

18. Filling in the Blanks….Mediators 18

19. Logic Model—Office of Workforce and Career Development Inputs Activities Outcomes Competent, sustainable, diverse workforce Improved health outcomes Conduct training Do career leadership development Evidence Base Do Strategic workforce planning …a miracle occurs…

20. Implicit Logic Model Activities Inputs Outcomes Workforce is Competent Sustainable Diverse Improved health outcomes Conduct training Do career leadership development Evidence Base Do Strategic workforce planning

21. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions Tangible products of activities

22. Pool (#) of eligible kids Pool (#) of screened kids Referrals (#) to medical treatment Pool (#) of assessed homes Referrals (#) for clean-up Lead Poisoning: Sample Outputs

23. Lead Poisoning: “Causal” Roadmap Activities Outcomes Outreach Do Environment Assessment ID Source and Refer for clean-up Lead Source Removed Train Families Screening ReducingEBLLs ImprovedDevelopmentand Intelligence Family performs in-home techniques Refer for Medical Treatment MedicalManagement ID kids with EBLL MoreProductiveand/or QualityLives Case Management 23

24. Pool (#) of screened kids (meeting likely risk profile) Pool (#) of eligible kids (with lead level >XXd/ul) Referrals (#) to (qualified or willing) medical treatment providers Pool (#) of assessed (“leaded”) homes Referrals (#) for clean-up (to qualified or willing orgs) Lead Poisoning: “Upgraded” Outputs: More than Simple Counts

25. Global Logic Model: Childhood Lead Poisoning Program Early Activities Outputs Later Activities Early Outcomes Later Outcomes (#) of eligible kids meeting risk profile (#) screened kids with lead < threshold (#) referrals to qualified medical tx (#) of families completing training (#) of “leaded” homes (#) referrals to qualified clean-up EBLL kids get medical treatment Family performs in-home techniques Lead source identified Environ cleaned up Lead source removed Do case mgmt Refer for medical tx Train family in in-home techniques Assess environ’t Refer house for clean-up EBLL reduced Develop’l slide stopped Quality of life improves Outreach Screening ID of elevated kids

26. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions Resource “platform” for the program

27. Funds Trained staff Legal authority to screen Relationships with orgs for med treatment and environmental clean-up Lead Poisoning: Sample Inputs

28. Lead Poisoning: “Causal” Roadmap Inputs Activities Outcomes Outreach Do Environment Assessment Funds Staff R’ships Legal Authority ID Source and Refer for clean-up Lead Source Removed ReducingEBLLs ImprovedDevelopmentand Intelligence Screening Family performs in-home techniques Train Families MedicalManagement ID kids with EBLL Refer for Medical Treatment MoreProductiveand/or QualityLives Case Management

29. Short-term Effects/ Outcomes Intermediate Effects/ Outcomes Long-term Effects/ Outcomes Inputs Activities Outputs Context Assumptions Moderators: Contextual factors that will facilitate or hinder (i.e. “moderate”) ability of activities to produce our outcomes

30. Moderators—Lead Poisoning • Political—“Hazard” politics • Economic— Health insurance • Technological— Availability of hand-held technology

31. Lead Poisoning: “Causal” Roadmap and Moderators Activities Outcomes Outreach Do Environment Assessment ID Source and Refer for clean-up Lead Source Removed Train Families Screening ReducingEBLLs ImprovedDevelopmentand Intelligence Family performs in-home techniques Refer for Medical Treatment MedicalManagement ID kids with EBLL MoreProductiveand/or QualityLives “hazard politics” Case Management insurance climate technology

32. Note! Logic Models make the program theory clear, nottrue!

33. Logic Models and Strategy and Planning • By Clarifying Sequence of Activities  Outcomes, Helps With… • Identifying/refining mission • Developing consensus on key intended outcomes—the “staked claim” • Enhances quality and clarity of BSC perspectives and objectives

34. Helpful Resources • NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: http://www.cdc.gov/eval/whatsnew.htm • Logic Model Sites • Innovation Network: http://www.innonet.org/ • Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/ • University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/ • CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/resources.htm#4 • CDC/STD: http://www.cdc.gov/std/program/progeval/TOC-PGprogeval.htm • Texts • Kellogg Foundation Logic Model Development Guide: www.wkkf.org • W.K. Kellogg Foundation Evaluation Resources: http://www.wkkf.org/programming/overview.aspx?CID=281 • Rogers et al. Program Theory in Evaluation. New Directions Series: Jossey-Bass, Fall 2000 • Chen, H. Theory-Driven Evaluations. Sage. 1990