Writing & Measuring Consumer Education & Referral Outcomes

1. Writing & Measuring Consumer Education & Referral Outcomes A QTS Web Training

2. Things to know • You should have the Power Point open on a computer in front of you • You have control of the power point on your own computer • The trainer will tell you when to go to the next slide

3. Conference Call Etiquette…. • Please mute your phone unless you have a question or comment • Please do NOT put your phone on hold during the call • If we should get disconnected by phone, call back in

4. Details . . . • Attendance will be taken at the conclusion of the call • If you need additional assistance, please work with either your: • Quality Assurance Training & Support Specialist, or • Regional Support Services Manager

5. What we will do… • Review expectations of 2.B.1.c and 2.C.1.a • Review outcome measurement basics • Discuss realities of outcome measurement • Anything else?

6. Results Accountability Guidebook Results Accountability Guidebook Child Care Resource Referral, Oregon State University Family Policy Program & Oregon Child Care Research Partnership, Pratt, Clara, and Weber Bobbie, 2000. Chapters 2, 3, and 6 http://www.naccrra.org/docs/results_accountability_guidebook_10-00.pdf. 

7. 2.B.1.c • An outcome based plan drives your services • Appropriate outcomes, indicators, activities, and inputs • Evaluation tool used to measure indicators • Compilation of that data (Based on what you are collecting, how do you know you’ve achieved the outcome?)

8. 2.C.1.a Outcome Based Evaluation • Tools measure progress toward outcomes • Surveying consumers (minimum 20% response rate) • Compiles and analyzes data • Uses survey results to inform services

9. Principles of Performance Measurement • Know what performance measurement is. • Begin with a program logic model. • Balance utility, feasibility, accuracy, and respect. • Identify the capacity for performance measurement. • Choose a design. • Use what is learned. Results Accountability Guidebook, pg. 81

10. Performance Measurement includes answering the following ?s: • What are the planned activities, outputs, and intended outcomes? • Does the program achieve its planned activities and outputs? • Does the program achieve its intended outcomes and results? Results Accountability Guidebook, pg. 82

11. Logic Model Results Accountability Guidebook, pgs. 18

12. Logic Model • Identify a goal • Identify one initial and one long term outcome • Identify the key program activities that will lead to the achievement of those outcomes • Specify the volume and quality of outputs • Identify the major resources or inputs needed • Realism check • Identify indicators of measures Results Accountability Guidebook, pgs. 18 & 19

13. Think of it in this way Outcome—what you want to happen; specific, observable, measurable, and achievable Activity—actions or strategies that are going to help you reach your outcome Indicator—measures qualitative change (outcomes) Outputs—measure activities in terms of volume or quality

14. What’s the difference? Outcome: Increased knowledge of child care options that meet family needs Activity: Use CCR&R database to link families with providers appropriate to family needs Outcome Indicator: 75% of parents responding to the follow up survey report an increase in knowledge of child care options that meet their needs (survey) Output Indicator: 256 families received referrals in the month of May. 97% of clients received between 5 and 10 child care options. (database)

15. What is an outcome? Answer this question: What is it you want to happen to clients who call your referral line? Think about how you measure if it has happened.

16. Outcomes are the program’s results for people & communities • Contribute to the achievement of longer term goals • Describe a desired state of knowledge, attitude, skills, behaviors, or circumstances • Outcomes are observable, measurable, and achievable Results Accountability Guidebook, pg. 24

17. Outcomes seek to… • Increase • Decrease • Maintain Results Accountability Guidebook, pg. 24

18. Sample Outcomes • Initial Increased knowledge of the characteristics of high quality child care • Longer term Increased family satisfaction with child care Results Accountability Guidebook, pg. 24

19. What is the difference? • Outcome • Output

20. What’s the difference? • Outcome Increased knowledge of the characteristics of high quality child care • Output 55 parents received quality indicator brochures in the month of December.

21. The difference • Outcome—states the desired change Increased knowledge of the characteristics of high quality child care • Output—volume and/or quality measure of what occurs as a result of activities. Volume: Number of families who received written materials describing characteristics of high quality child care Quality: Parent satisfaction with consultations and educational materials about quality child care characteristics

22. Another example • Outcome Increased knowledge of child care options that meet family needs • Output Volume: Number of families who received child care referrals Quality: Accuracy of referrals received by families

23. Performance Measurement • Compares planned activities, outputs, and outcomes to: • Actual accomplishments • Accomplishments of other programs • Population norms • Uses simple design for data collection • Includes all or a good sample of participants

24. Performance Measurement (continued) • Uses records, observations, surveys • Relies on descriptive data analysis • Informs service delivery and demonstrates results Results Accountability Guidebook, pg. 83

25. Decide what you will measure “Not everything that is measurable is important; not everything that is important is measurable.” Michael Patton, Evaluator Results Accountability Guidebook, pg. 88

26. How do you decide what to measure? • What info is most needed to improve program operations and outcomes? • What will produce the most accurate information? • What is feasible given staff and participant time, staff skills, and program resources? • What is worthwhile and respectful of participants and staff? Results Accountability Guidebook, pg. 89

27. Identify your capacity • Level A • Level B • Level C Results Accountability Guidebook, pg. 90

28. In deciding what to measure, the so that chain can help Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that Outcome 1: Families have increased knowledge of child care options that meet their needs, so that Outcome 2: Families have increased access to child care options that meet their needs, so that Outcome 3: Families use information to find child care, so that Outcome 4: Families are satisfied with child care arrangement, so that Outcome 5: There is an increased probability of continuity of care for the child, so that Ultimate goal: Increased family sufficiency and ability to achieve personal goals is achieved.

29. Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that Outcome 1: Families have increased knowledge of child care options that meet their needs, so that Outcome 2: Families have increased access to child care options that meet their needs, so that Outcome 3: Families use information to find child care, so that Outcome 4: Families are satisfied with child care arrangement, so that Outcome 5: There is an increased probability of continuity of care for the child, so that Ultimate goal: Increased family sufficiency and ability to achieve personal goals is achieved. Key Activity: Provide information about the characteristics of high quality child care, so that Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that Outcome 3: Parents choose high quality child care arrangement, so that Outcome 4: Parents are satisfied with child care arrangement, so that Outcome 5: There is an increased probability of continuity of care for the child, so that Ultimate goal: Families experience increased family sufficiency and ability to achieve personal goals is achieved. What is it you notice as you go further up the chain?

30. What is feasible to measure? Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that Outcome 1: Families have increased knowledge of child care options that meet their needs, so that Outcome 2: Families have increased access to child care options that meet their needs, so that Outcome 3: Families use information to find child care, so that Outcome 4: Families are satisfied with child care arrangement, so that Outcome 5: There is an increased probability of continuity of care for the child, so that Ultimate goal: Increased family sufficiency and ability to achieve personal goals is achieved.

31. And these? Key Activity: Provide information about the characteristics of high quality child care, so that Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that Outcome 3: Parents choose high quality child care arrangement, so that Outcome 4: Parents are satisfied with child care arrangement, so that Outcome 5: There is an increased probability of continuity of care for the child, so that Ultimate goal: Increased family sufficiency and ability to achieve personal goals is achieved.

32. At the very least… …agencies should track their initial outcomes of key activities.

33. Choose a design • When will information be collected? • From whom will information be collected? • What information will be collected and how will it be analyzed? • How will participants’ needs, rights, and welfare be protected? Results Accountability Guidebook, pg. 92

34. Most common • Database • Surveys • Written • Telephone

35. Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that Outcome 1: Families have increased knowledge of child care options that meet their needs, so that Outcome 2: Families have increased access to child care options that meet their needs, so that Outcome 3: Families use information to find child care, so that Key Activity: Provide information about the characteristics of high quality child care, so that Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that Outcome 3: Parents choose high quality child care arrangement, so that How does your survey help measure your outcome?

36. Outcome 1: Families have increased knowledge of child care options that meet their needs, so that Outcome 2: Families have increased access to child care options that meet their needs, so that Outcome 3: Families use information to find child care, so that Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that Outcome 3: Parents choose high quality child care arrangement, so that Were our services helpful?

37. Outcome 1: Families have increased knowledge of child care options that meet their needs, so that Outcome 2: Families have increased access to child care options that meet their needs, so that Outcome 3: Families use information to find child care, so that Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that Outcome 3: Parents choose high quality child care arrangement, so that Did you find child care?

38. An example Outcome 2: Families have increased access to child care options that meet their needs Indicator: 85% of parents report that CCR&R consultation helped them to find appropriate child care arrangements, even if the setting found was not one of CCR&R referrals Sample Survey Question: Did you eventually use one of the providers referred to you by the CCR&R for this child? Yes No If yes, are you still using this provider? Yes No How long? If no, What did you eventually do? Give an array of choice Did your contact with the CCR&R help you make a more informed choice when selecting child care for your child? Yes No

39. Another example Outcome 1: Parent knowledge of characteristics of high quality child care increases Indicator: 85% of parents responding to the survey report an increase in their knowledge of the characteristics of high quality child care as a result of their contact with the CCR&R Sample question: How would you rate your current understanding of things to consider in choosing good quality child care? 1 2 3 4 5 N/A How would you rate your understanding of these things before you talked with the CCR&R? 1 2 3 4 5 N/A

40. How could you measure these? • Increased usage of referral services by families seeking child care and increased public awareness of the CCR&R services • Increased usage of quality indicators when choosing child care

41. Key points • Start with a logic model • What do you want to accomplish? (outcome) • How will you know if it has happened? (indicator) • Proceed realistically • Choose a design • Use what is learned

42. Implement these strategies to avoid common errors… • Understand terminology • Areas of particular confusion outcomes vs. activities; outcomes vs. outputs; indicators vs. outputs • Include only outcomes that are realistically measurable and that you are or will be measuring • Make sure the tool that you are using is actually measuring what you say it is • Include a compilation of data that supports achievement of the outcome

43. What we wanted to do • Review expectations of 2.B.1.c and 2.C.1.a • Review outcome measurement basics • Discuss realities of outcome measurement

44. Wrapping up Attendance • Please STAY ON the line until everyone’s name is called, otherwise we hear only beeps! • If your name is NOT called and/or you are sharing the connection with someone else, please email me at Gail.Upton@naccrra.org with: • Your name, CCR&R name, position title, e-mail address, and Agency ID

45. Questions: Gail.Upton@NACCRRA.org OR (866-782-2261) Contact Info