High Quality Performance Measures

1. High Quality Performance Measures • What is Performance Measurement? • What makes a high quality performance measure?

2. Learning Objectives • By the end of this module, you will be able to: • Describe basic performance measurement concepts • Describe how to strengthen performance measures through: • Alignment with theory of change • Alignment of outputs and outcomes • Selection of a meaningful outcome

3. Module Overview • Performance Measurement • Concept and definitions • Comparison to impact evaluation • Strengthening Performance Measures through: • Alignment with the Theory of Change • Alignment of outputs and outcomes • Selection of a meaningful outcome

4. What is Performance Measurement? • Systematic process of regularly measuring: Outputs • Amount of service provided (e.g. people served, products created, or programs developed) Outcomes • Reflect the changes or benefits that occur as the result of the service • Can reflect changes in individuals, organizations, communities, or the environment • Address changes in attitudes/beliefs, knowledge/skills, behavior, or conditions

5. Performance Measurement and Impact Evaluation • Similarities • Systematic data collection • Measurement of progress • Collect data on: • What took place (intervention) • Outputs (amount of service) • Outcomes (changes)

6. Performance Measurement and Impact Evaluation

7. Performance Measurement and Impact Evaluation (continued) *Capable of producing valid (accurate) results.

8. Learning Exercise

9. Learning Exercise

10. Ensuring Quality Planning • Performance Measure • Intervention • Output • Outcome Align output and outcome Align elements within ToC • Theory of Change (ToC) • Problem/Need • Intervention • Outcome Review for meaningful outcome

11. Performance Measurement and Alignment • Alignment in Performance Measurement: Logical flow and connection between the different elements • Alignment within the theory of change • Alignment between output and outcome

12. Alignment within Theory of Change Community Need/Problem Specific Intervention “cause” Intended Outcome “effect” • Evidence • Guides choice of intervention • Supports cause-effect relationship A theory of change defines a cause-and-effect relationship between a specific intervention and an intended outcome.

13. Mentoring Example*: Alignment within Theory of Change Which outcome is aligned with the intervention? Students participate in the mentoring program Students increase academic performance Students improve school attendance NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills. OUTCOME: ________ *Descriptions in examples are abbreviated.

14. Mentoring Example: Alignment within Theory of Change Which outcome is aligned with the intervention? Students participate in the mentoring program Students increase academic performance Students improve school attendance NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills. OUTCOME: ________

15. Mentoring Example: Alignment within Theory of Change NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills. OUTCOME: Students improve school attendance.

16. Learning Exercise NEED:35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Outcome Veterans report increased confidence in using the internet for a job search. Veteran family member report completion of skill classes. Veterans find or are placed in jobs. • Intervention • Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches. • Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. • Military families receive job placement counseling.

17. Learning Exercise (continued)Select the outcome. NEED:35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Outcome Veterans report increased confidence in using the internet for a job search. Veteran family member report completion of job skills classes. Veterans find or are placed in jobs. • Intervention • Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches. • Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. • Military families receive job placement counseling.

18. Learning Exercise (continued)Select the intervention. NEED:35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Outcome Veterans find or are placed in jobs. • Intervention • Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches. • Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. • Military families receive job placement counseling.

19. Learning Exercise (continued)Summary: Alignment within Theory of Change NEED:35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Outcome Veterans find or are placed in jobs. • Intervention • Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.

20. Alignment of Outputs and Outcomes • Output results from the intervention • Outcome is likely to result from output • Output and outcome measure the same intervention and beneficiaries

21. Housing Example:Output-Outcome Alignment Do the intervention and output align? Yes No Why or why not? INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs). OUTCOME: Construction team members improve construction skills.

22. Housing Example:Output-Outcome Alignment Do the intervention and output align? Yes No Why or why not? Logical to assume if well-designed and implemented INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs). OUTCOME: Construction team members improve construction skills.

23. Housing Example:Output-Outcome Alignment Do the output and outcome align? Yes No Why or why not? INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs, and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing services (upgrades and repairs). OUTCOME: Construction team members improve construction skills.

24. Housing Example:Output-Outcome Alignment Do the output and outcome align? Yes No Why or why not? 2 different groups of beneficiaries INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs). OUTCOME: Construction team members improve construction skills.

25. Learning ExerciseOutput-Outcome Alignment INTERVENTION:National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months. Outcome Children demonstrate gains in school readiness in terms of literacy skills. Children demonstrate gains in school readiness in terms of numeracy (math) skills Teachers implement new math curriculum in the classroom. • Output • Children complete participation in the early childhood program. • Teachers complete training with expert math teachers. • Children complete participation in preschool art group.

26. Learning ExerciseOutput-Outcome Alignment INTERVENTION:National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months. Outcome Children demonstrate gains in school readiness in terms of literacy skills. Children demonstrate gains in school readiness in terms of numeracy (math) skills Teachers implement new math curriculum in the classroom. • Output • Children complete participation in the early childhood program • Teachers complete training with expert math teachers. • Children complete participation in preschool art group.

27. Learning ExerciseOutput-Outcome Alignment INTERVENTION:National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months. Outcome Children demonstrate gains in school readiness in terms of literacy skills. Children demonstrate gains in school readiness in terms of numeracy (math) skills Teachers implement new math curriculum in the classroom. • Output • Children complete participation in the early childhood program .

28. Examples of Aligned Output-Outcome National Performance Measures National performance measures at the Resource Center: www.nationalserviceresources.org/npm/home

29. Identifying a Meaningful Outcome • The Outcome should: • Be central to your theory of change • Match the type of change (attitude, knowledge, behavior, or condition) you want to measure • Be ambitious but manageable

30. Central to the Theory of Change • How is the proposed outcome supported by the theory of change? • Evidence for intervention should support the outcome • Evidence supports cause-and-effect relationship between intervention and outcome

31. Central to the Theory of Change • How is the proposed outcome central to the theory of change? • Addresses the need/problem • Likely change given the intervention • A key, not preliminary or peripheral, outcome

32. Mentoring Example: Central to Theory of Change Which outcome is central? Families are satisfied with the mentoring program Students improve engagement with peers Students improve school attendance NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills.

33. Type of outcome: Accurate and Clear • Is the outcome getting at what you wish to measure? • Articulates the question you really want to answer • Has the outcome type been identified: attitude, knowledge/skills, behavior or condition? • Outcome type should be same as theory of change • Outcome type will inform instrument

34. Veteran Example: Type of Outcome Which type of outcome? Veterans report increased confidence about finding employment. (attitude) Veterans demonstrate new technical skills. (knowledge/skills) Veterans are placed in jobs. (condition) • NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications… INTERVENTION: National service participants support veterans in completing training programs by assisting in locating appropriate programs, securing financial aid, and by providing tutoring resources and internship placements.

35. Ambitious and Manageable • Is the outcome meaningful AND manageable? Consider: • Staff time and expertise • Reporting constraints

36. Ambitious and Manageable • Is the outcome too modest? Consider: • Outcome should reflect the main change expected from the intervention • Is the outcome too ambitious? Consider: • Program timeframe • Scope of the intervention • Severity of the problems being addressed

37. Mentoring Example: Ambitious and Manageable Which outcome is meaningful and manageable? Student increase outdoor teamwork skills Students increase graduation rates from high school Students improve school attendance NEED: Documented high rates of chronic absence from school… INTERVENTION: National Service participants provide individual and group mentoring to 6th graders with documented attendance problems…

38. Mentoring Example: Ambitious and Manageable Which outcome is meaningful and manageable? Student increase outdoor teamwork skills Students increase graduation rates from high school Students improve school attendance NEED: Documented high rates of chronic absence from school… INTERVENTION: National Service participants provide individual and group mentoring to 6th graders with documented attendance problems…

39. Learning Exercise Outcomes Students know location of the library Student improve reading ability Students graduate from high school Students increase school attendance Intervention: Year long mentoring project with 4th graders– literacy focus. • Which outcome is too ambitious? (select one) • Which outcome is too modest? (select one) • Which outcomes look ambitious but manageable? (select two)

40. Learning Exercise Outcomes Parents are interested in learning more about the program. There is reduced gang violence in the area. Donations for sustaining activities at the two sites are secured. Trained community volunteers are serving at the new sites. Intervention: An afterschool program for children ages 6-12 is expanded to two new sites. • Which outcome is too ambitious? (select one) • Which outcome is too modest? (select one) • Which outcomes look ambitious but manageable? (select two)

41. Learning Exercise Outcomes Youth develop long-term healthy eating habits. Youth increase healthy food intake. Youth improve cooking skills. Youth are motivated to eat more vegetables. Intervention: 6-month nutrition and health program for high school students. • Which outcome is too ambitious? (select one) • Which outcome is too modest? (select one) • Which outcomes look ambitious but manageable? (select two)

42. Learning Exercise Outcomes Youth increase knowledge of native stream bank plant species. 250,000 residents in the county benefit from restored watershed. Youth increase interest in protecting the environment. Water quality is improved over 6 mile clean-up area. Intervention: Year-long riparian (river) restoration project involving youth volunteers. • Which outcome is too ambitious? (select one) • Which outcome is too modest? (select one) • Which outcomes look ambitious but manageable? (select two)

43. Learning Exercise • The most meaningful and manageable outcome for this intervention is: • Number of individuals satisfied with quality of the work. • Number of individuals securing mortgage loan approval for house purchase within 5 years. • Number of individuals transitioned into safe housing. INTERVENTION: National Service participants assess the housing up-grade requests from the community members and address dilapidated, unsafe or inaccessible housing. They plan and then execute housing remodels, upgrades and repairs for economically disadvantaged individuals and individuals with disabilities.

44. Summary of Key Points • Performance measurement is a systematic process of measuring progress (outputs and outcomes) • Performance measurement does not seek to “prove” a theory of change, but can provide snapshots. • Impact evaluation can determine if results occurred because of the intervention. • Strong performance measures align with the theory of change (need, intervention and outcome) • The intervention is based on evidence that supports a cause-and-effect relationship between the intervention and an intended outcome. • The theory of change helps identify an appropriate outcome to measure.

45. Summary of Key Points • Strong performance measures are aligned • Output results from the intervention • Outcome is likely to result from output • Output and outcome measure the same intervention and beneficiaries • Strong outcomes should: • Be central to the theory of change • Match the type of change you want to see (attitude, knowledge/skills, behavior, condition) • Be ambitious but manageable

46. Resources • CNCS Priorities and Performance Measures: http://www.nationalservice.gov/resources/npm/home • Program Specific Notices of Funding Opportunities and Application Instructions: http://http://www.nationalservice.gov/for_organizations/funding/nofa.asp • Link to Evaluation