Evaluating Program Success

1. Evaluating Program Success Cherie McCraw Born to Read Institute November 2001

2. Plan for Evaluation • Decide what is appropriate • Decide what is feasible • Decide what is adequate • Don’t reinvent the wheel “Evaluate with Passion and Purpose”

3. TraditionalProject Measurement Inputs Activities Outputs Resources - Funding - Staff - Volunteers - Equipment Services - Training - Tutoring - Mentoring - Installing Products - # of classes - # of students - # of books - # participants

4. We don’t just look at “countable” things anymore. We look at the outcome(s) of our programs… how are people changed? how are people benefited?

5. Outcomes Measurement Inputs Activities Outputs Outcomes Change in Client - Knowledge - Skills - Behavior - Condition Resources - Funding - Staff - Volunteers - Equipment Services - Training - Tutoring - Mentoring - Installing Products - # of classes - # of students - # of books - # participants

6. Pressure to Know More • From: • Funders - local, state and federal • Public - press, community “How do we know the dollars make a difference?” • From: • Boards • Management • Staff “How do we best use our scarce resources?”

7. Why Change? • The numbers don’t answer “So what?” • Outcomes give us structure • Create a strong case for funds in a competitive market • Establish ourselves as players for community-wide outcomes • “Just because we always have” …is not a good reason to do something today

8. Outcomes Can: • Provide focus • Measure results • Take 3-5 years to implement

9. Outcomes Can’t: • Tell you whether your program caused the outcome • Indicate why the level of outcome was achieved • Suggest actions to take to improve the outcome

10. Choosing Which Outcomes to Measure • Use the KISS method! • Look at the data you already collect • Choose outcomes that are clear and meaningful to the public • For similar projects or grants- identify common outcome(s)

11. Resources money staff volunteers equipment & supplies Constraints laws regulations funders’ requirements Outcome Measurement Inputs Activities Outputs Outcomes

12. Criteria for Inputs • Identify a resource used for the project • Identify a single element • Are concise, clear, and lingo-free • Are quantified if possible

13. Easy books Children’s librarian Partners: (ID each) Even Start families Deposit collections 5 trained volunteers 2 library computers Printed materials from hospitals 51 Laubach Literacy Action trained & certified volunteers Library facilities Matching funds Grant funds Office supplies 1 County vehicle BTR program materials Examples of BTR Inputs

14. Services training education counseling mentoring Outcomes Measurement Inputs Activities Outputs Outcomes

15. Criteria for Activities • Are verbs • Are focused on the client not on the operations of the program • Show what the program does • Identify a single action

16. Provide weekly toddler and infant storytimes in all library locations Put packets together Present parenting workshops Conduct lapsits Provide bags for parents Train volunteers Enhance collections with age appropriate books Distribute packets through home visits Enroll low income new parents/teens and their children Provide referral service to the adult literacy tutoring program Examples of BTR Activities

17. Develop brochure Conduct BTR publicity campaign: radio, billboards, and posters Develop a BTR Family Literacy web page with links Conduct phone interviews, personal interviews and focus groups Host meetings with partners Implement a summer reading program Create BTR newsletter and mailing list Issue library cards to program participants Examples of BTR Activities

18. Products classes taught counseling sessions conducted educational materials distributed hours of service delivered participants served Outcomes Measurement Inputs Activities Outputs Outcomes

19. Criteria for Outputs • Measure “how many” for the activities • Often are simply the activities quantified-50 computer classes taught • Measure a single activity • Do not measure a change for a client (that would be an outcome)

20. 120 pregnant new moms served 8 lapsits at Health Department 12 Expectant Mothers programs, 150 people attend 5,000 packets 50 new books added 2 training sessions for project partners # attending lapsits # of parent programs given # of bags distributed # of volunteers # web page hits # of posters, billboards, radio PSAs # parents/children attending storytimes Examples of BTR Outputs

21. Benefits for People new knowledge increased skills changed attitudes or values modified behavior improved condition altered status Outcomes Measurement Inputs Activities Outputs Outcomes

22. Criteria for Outcomes • Are client-focused • Measure a single meaningful change in the client • Are within the scope of the project • Identify who achieves the outcome • Are concise, clear, and lingo-free • Are objective • Are specific

23. Parents or caregivers read to their children Parents set a goal for their own or their child’s education BTR staff gain insight into the information needs of the clients Community organizations cooperate to provide services to break inter-generational cycle of illiteracy Parents or caregivers use the library Examples of BTR Outcomes

24. We know what we want to achieve with our programs… BUT... How do we measure our success AND demonstrate to funders, boards and commissions that we were successful?

25. Your Evaluation Plan!

26. Outcomes Measurement:Evaluation Plan Inputs Activities Outputs Outcomes Indicators Sources/Methods Data you collect to measure indicators of success will let you know you have achieved your outcome(s) and to what extent you have achieved them.

27. Criteria for Indicators • Show how well a program is doing on an outcome • Show that the outcomes have been achieved- “Number of parents who read daily to their children” • Are stated as a number and/or a percentage • Each outcome has at least one indicator • Measure at the individual, not group, level

28. Setting Targets • Most programs cannot establish targets in the first year of collecting data • Possible sources for targets: • Program baseline • National statistics • Best practices research

29. 40% program participants come to the library. 50% increase in time parents or caregivers read to their children. 150 library cards issued. # of times library card is used per month. # of positive responses to questionnaires. # of parents sharing books and reading with their children. Examples of BTR Indicators

30. # & % of teen program participants who use the library. # & % of parents reading to their child on a regular basis. # & % of parents attending workshops. # & % of parents setting educational goals. # & % of partners who say the partnership helped extend their services. # of certificates given at end of program. Examples of BTR Indicators

31. Criteria for Sources • “Who” provides the data • There should be at least one source for each indicator • Possible sources include: • Clients • Family members • Project staff • Existing records • Volunteer observers

32. Criteria for Methods • “How” the outcome will be measured • Provide specific data to measure an indicator • Possible methods include: • Review of project records • Questionnaire or survey • Interview • Rating by a trained observer

33. Library staff/Count of coupons for diaper bags given at programs and redeemed at the library. Parent interviews by library and Healthy Start staff. Automated circulation system/track of usage of coded library cards. Program administrator/Attendance records. Questionnaires distributed by home visitors. Examples of BTR Data Sources and Methods

34. Reading diaries checked by Library staff. Project manager/Student information forms. Project manager/agreement forms signed by students and tutors Project manager/Focus group with program participants. Library staff/Telephone survey asked of BTR participants. Tutors/Pre- and post-tests. Examples of BTR Data Sources and Methods

35. Plan Before Collecting Data • Train data collectors with the instruments they will be using or giving out. • Develop procedure for accurately recording the results. • Setting the scene: • Signage • Collection boxes • Return envelopes

36. Uses of Outcome Findings • Internal • Provide direction for staff • Identify training needs • Improve programs • Support planning • Guide budgets and justify resource allocations • Suggest outcome targets

37. Uses of Outcome Findings • External • Recruit talented staff and volunteers • Promote the program to potential participants and referral sources • Identify partners for collaboration • Enhance the program’s public image • Retain and increase funding

38. The most compelling argument you make will occur when you convey the passion you have for your project!