Using Logic Models to Plan, Monitor, and Evaluate Programs

1. Using Logic Models to Plan, Monitor, and Evaluate Programs Erin B. Comartin, Ph.D., LMSW School of Social Work Wayne State University Adapted from training held by: Joanne Sobeck, Ph.D.

2. Agenda • Program development • Getting to outcomes • A definition of a logic model • An overview of the basic uses • Defining the elements of a logic model • Working through the development of logic model together • Challenges in logic modeling

3. Program Development • A process for defining, refining, and confirming core elements of a program Step 1: Specify the problem/issue and develop program theory Step 2: Create and/or revise program materials Step 3: Refine and confirm program components Step 4: Determine effectiveness in various settings and groups Step 5: Decide on expansion

4. Planning What is the problem? What are the needs? What do we need to change? Outcome Evaluation Is it effective? Goals & Objectives What do we want to achieve? Implementation How do we want to do it? Getting to Outcomes... Process Evaluation Are we doing what we planned?

5. Getting to Outcomes by way of Logic Models • Program outcomes are a reflection of a translational process whereby assumptions about needs and theories-in-action are examined, goals are established, and activities are planned all within an environment of existing and potential resources.

6. A Logic Model Defined A logic model sets out how an intervention (such as a project, a program, or a policy) is understood or intended to produce particular results. • Rogers, P.J. (2005). Logic models. In S. Mathison (Ed.), Encyclopedia of Evaluation. Beverly Hills, CA: Sage Publications.

7. Logic Models Are Visual—You Can See Them • Tables • Figures/Graphs

8. Logic Models Can Be Simple or Complex Great Outcomes Our Program • Simple • Complex

9. Logic Models Express a Program Theory – Also called change process, theory of change • How the pieces fit together; expectation of how the program comes together resulting in positive outcomes Inputs → Throughputs → Outputs

10. What are logic models used for?

11. Uses • Clarifying what is really intended • Enhancing communication among team members • Managing projects • Designing evaluation plans • Documenting a project and how it worked

12. 1. Clarification • Do our activities really connect to our goals?

13. 1. Clarification • Are our outcomes specified precisely? You are here

14. 2. Enhancing communication • The same words and situations have different meanings for different people.

15. 2. Enhancing communication • What is academic achievement? • Class grades • MEAP scores • Homework turned in on time • Student’s perception of “doing well in school” • Parent’s perception of “doing well in school”

16. 3. Managing Projects • With the addition of time lines or work plans, logic models can aid project managers in keeping on task and on track

17. Sample Logic Model Regional Work Plan Goal 1: To prevent alcohol-related traffic crash deaths & related high risk drinking behaviors and consequences

18. 4. Designing Evaluation Plans

19. School, program, and community context Suspensions & expulsions will each be reduced by 10% per year. Student absenteeism will be reduced by 5% per year. PROGRAM IMPLEMENTATION IMMEDIATE OUTCOMES INTERMEDIATE OUTCOMES ULTIMATE OUTCOMES Graduation rates will increase by 10% by 2006. Have identified theory of change/goals Academic enrichment represents at least 40% of the program. Utilization of programs Increased academic engagement Provide academic enrichment for students with learning disabilities. Improved academic outcomes 90% of the students will receive tutoring and homework assistance at least 2 days/wk. Curriculum/ activities tied to goals Engagement with staff Increased valuing of education The # of students with a B- or better will increase by 5%. Of the regularly attending students, 85% will increase their math scores by 5%. 85% of parents will attend program activities and meetings for their child. Sites will host at least 5 projects directly linked with the school curriculum. Collaboration with school and community Family support of involvement Increased academic skills Mastery of new computer software. Increase frequency and enjoyment of independent reading. 75% of participating parents will demonstrate they have learned positive practices to support academic achievement. Sufficient staffing/ training Linkages within school Improved socio-emotional functioning Improved behavioral outcomes Teacher surveys will have a 70% higher satisfaction rate with the academic consultation process. Recruitment/ retention processes Individual and family characteristics Incentives for academic achievement will be established at each site.

20. 5. Documenting a project and figuring out how it worked (what did and did not work) Trainers Better client outcome Training In-Service All staff attend Increased knowledge New practices Training Curriculum

21. So across the life of a program… • Program development and planning: • Planning tool to develop program strategy • Provide way to explain how’s and why’s to stakeholders/funders • Develop common language and expectations among team members • Identify indicators and measures for outcome assessment • Program implementation: • Provides management plan • Program evaluation and reporting: • Assess progress toward goals • Assess breakdown points in process or places where the model doesn’t work

22. Let’s start with the basics…

23. Elements of a Logic Model • Assumptions: Perceptions of need and problems for the target population • Goals: Long term impacts; desired state • Outcomes: Changes that occur in people (or something) as a result of the program • Outputs: The accomplishments, products or service units of a program • Activities: The services of a program • Inputs: Resources a program uses to carry out its activities

24. Elements in Detail • Assumptions • Effective programs and logic models are built on correct assumptions about the needs of clients. This requires research and thoughtful assessment.

25. Elements in Detail • Goals • This is your broad vision for the future. It is your vision for the target population if the problem did not exist. What conditions do you see for the target population?

26. Elements in Detail • Outcomes • Outcomes are the results you can expect. Limited in time, single end result. Identify results to be achieved rather than activities to be performed.

27. Elements in Detail • Outputs • Outputs are the results of your service activities. They have both a client and service focus. The client focus identifies the number of people served, whereas the service focus identifies the number and length of sessions.

28. Elements in Detail • Activities • What do we have to do to insure our goals (short term mainly) will be met? Who will do what, with whom, when and how will you know its done? These are the methods that will take you to your goals.

29. Elements in Detail • Inputs • What are the basics needed to accomplish our goals? Resource planning. Specific expertise (know your subject), physical labor (staff, time commitment), community support (volunteers, resource donations, community impact- acceptance) required materials (facility).

30. Logic Model Template

31. Graph Logic Models To Plan and Evaluate the Program, Think Backward-- • Why do we do what we do? • What do you hope will happen in the long run? • What are the expected results? • What will you do to make those things happen? • What resources do you need? • But implement forward!!!

32. A Logic Model Template (one of many)

33. Logic Model Elements

34. Logic Model Elements

35. Logic Model Elements

36. Logic Model Elements

37. Logic Model Elements

38. Logic Model Elements

39. An inadequate logic model…

40. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes Parent training More responsive children

41. A miracle occurs….

42. Models that do not take into account the middle steps… • Have outcomes that are far away; the farther the outcome, the more it can be affected by other factors besides your program • Do not provide you with information about where the program might not have been effective • Do not give you the opportunity to present data about where your program did work Let’s try again…

43. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes More responsive children

44. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes Children will become more responsive if they have regular, appropriate interaction Parents interact regularly/ frequently with children Parents interact at appropriate level with children More responsive children Parents use techniques to focus children’s attention

45. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes Parents will interact regularly if they understand it can make a difference Parents interact regularly/ frequently with children Parents understand need for intensive interaction Parents interact at appropriate level with children More responsive children Parents learn strategies for interaction Parents use techniques to focus children’s attention Parents will interact appropriately if they know what to do

46. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes Parents interact regularly/ frequently with children Parents who participate in all the home visits and get video feedback will understand how interaction is important and will learn what to do Families receive monthly home visits Parents understand need for intensive interaction Parents interact at appropriate level with children More responsive children Parents learn strategies for interaction Parents receive video feedback at all visits Parents use techniques to focus children’s attention

47. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes This home visiting program is what’s needed to train these parents Parents interact regularly/ frequently with children Families receive monthly home visits Parents understand need for intensive interaction Provide parent education program thru home visits Parents interact at appropriate level with children More responsive children Parents learn strategies for interaction Parents receive video feedback at all visits Parents use techniques to focus children’s attention

48. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes This program can work if it has trained and supervised home consultants and a proven curriculum Parents interact regularly/ frequently with children Trained home consultants Families receive monthly home visits Parents understand need for intensive interaction Provide parent education program thru home visits Parents interact at appropriate level with children More responsive children Evidence-based curriculum Parents learn strategies for interaction Parents receive video feedback at all visits Parents use techniques to focus children’s attention Regular supervision

49. Intervention for Autistic Children Inputs Activities Outputs Initial outcomes Intermediate outcomes Long-term outcomes Parents interact regularly/ frequently with children Trained home consultants Families receive monthly home visits Parents understand need for intensive interaction Provide parent education program thru home visits Parents interact at appropriate level with children More responsive children Evidence-based curriculum Parents learn strategies for interaction Parents receive video feedback at all visits Parents use techniques to focus children’s attention Regular supervision

50. Parent Training Program for AutismThis also becomes your blueprint for evaluation indicators