Performance Measures AmeriCorps Project Director Training Saratoga, NY

1. Performance Measures AmeriCorps Project Director Training Saratoga, NY October 7th – 9th, 2013

2. AGENDA • Theory of Change • Activity • Performance Measure Basics Overview • Activity/Questions

3. Theory of Change

4. Theory of Change Perspective INTRODUCTION • Looks at cause and effect relationships • Identifies specific interventions to achieve the desired result • Shifts thinking from “what are we doing” to focus on “what do we want to achieve”

5. Overview of Prohibited Activities Theory of Change Elements STATISTICS documenting the need • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship

6. Prohibited Activities Everyday Life Example • I have strep throat (problem) • I will take antibiotics (intervention) • I will get better (outcome) • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship

7. Prohibited Activities Everyday Life Example cont. • EVIDENCE • Penicillin • Amoxicillin • Tetracycline • There are many antibiotics out there • Which antibiotic do I take for strep? • Look at evidence to make the choice

8. Overview of Prohibited Activities Theory of Change Elements STATISTICS documenting the need • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community?

9. Overview of Prohibited Activities Theory of Change Elements - Community Need • Data documenting problem/need should answer these questions: • SCOPE: Who and how many are directly affected? How severe is this? • SIGNIFICANCE: What makes this a compelling need? Is it likely to become worse? What will happen if we do nothing? • CAUSE(S): Why does the need exist? How is it perpetuated?

10. Overview of Prohibited Activities Theory of Change Elements - Statistics Data documenting problems/needs should be statistic that document the extent and severity of the problem. STATISTICS documenting the need • The best data comes from reputable primary sources like government agencies, institutions, and universities that do their own research that is: • Locally relevant • Up-to-date • Note: New reports are not primary sources but could be used as back-up to primary evidence.

11. Overview of Prohibited Activities Theory of Change Elements - Outcome STATISTICS documenting the need • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship What change are you hoping to make related to the identified need?

12. Overview of Prohibited Activities Theory of Change Elements - Outcome Identifying the Intended Outcome. • Healthy Futures Program Example • Possible outcomes to measure: • Increased knowledge of what is “healthy food” • More frequent choice of healthy foods to eat • More frequent involvement in physical activity or exercise • Improved physical condition

13. Overview of Prohibited Activities Theory of Change Elements - Intervention STATISTICS documenting the need • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome?

14. Overview of Prohibited Activities Theory of Change Elements - Intervention Describe the design and dosage of your intervention (service activity): • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship • Design – who does what with whom? • Dosage • Frequency – how many sessions a week? • Intensity – length of each session. • Duration – how many total weeks of sessions?

15. Overview of Prohibited Activities Theory of Change Elements - Intervention Healthy Futures Program Example • Design: national service participants implement the Shape Up curriculum with economically disadvantaged urban girls ages 14-16 to increase physical activity (30 minutes/session) and educate them on healthy eating • Frequency: twice a week afterschool • Intensity: 60 minutes per session • Duration: 12 weeks

16. Overview of Prohibited Activities Theory of Change Elements - Evidence • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship • Evidence demonstrates that the proposed intervention (design, dosage) is likely to lead to the outcome. • Service activities that are supported by research to demonstrate a high likelihood that these activities will address the need, and result in intended changes.

17. Overview of Prohibited Activities Theory of Change Elements - Evidence • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship EVIDENCE: Information that supports your choice. It is NOT enough to just say “we believe” our intervention is “likely” to be successful. • Evidence provides a reality check for theory of change • New programs: What is the recommended design (specific program activities) and dosage (frequency, intensity, and duration) to achieve an intended outcome? • For existing programs: Is there sufficient evidence for the intervention to continue its use? Based on the evidence, are there modifications to the intervention that should be considered to have greater impact? Do you need to choose a new intervention?

18. Overview of Prohibited Activities Theory of Change Elements - Evidence • EVIDENCE • Guides choice of intervention • Supports cause-effect relationship Possible sources of evidence: • Your past performance measurement outcome data • Results from an impact evaluation of your program • Research that documents the outcomes of similar programs • Impact evaluations that document outcomes of similar programs

19. Overview of Prohibited Activities Testing Your Theory of Change Is your Theory of Change: • PLAUSIBLE: Does the logic of the model seem correct: “if we do these things, will we get the results we expect?” • FEASIBLE: Are resources sufficient to implement the chosen intervention? • MEANINGFUL: Are intended outcomes important? Is the magnitude of expected change worth the effort?

20. Overview of Prohibited Activities Theory of Change Summary Summary of Key Points: • A theory of change identifies cause/effect • The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence • Data documenting community need should show scope, significance, and causes

21. Case Study

22. Performance Measure Overview

23. Overview of Prohibited Activities Performance Measures Why Measure Performance? • Accountability to funders and stakeholders • Tell your story, justify funding • Recognizing progress; measuring change • Reliable information collected in a systematic way • Program improvement • Spot and correct problems • Strengthen the intervention • Determine where to allocate limited resources

24. Overview of Prohibited Activities Performance Measures Using Performance Measurement in Program Management • Strategy and Planning • Use for planning and program development • Determine what you hope to achieve at different time periods • Develop site applications and Memoranda of Understanding templates (If working with sites) • Consider how partnerships will work to achieve outputs and outcomes • Site Recruitment and Selection • Define expectations • Assess site applications; determine which sites are best able to implement intervention

25. Overview of Prohibited Activities Performance Measures Using Performance Measurement in Program Management continued • Program Implementation • In conversations with staff, sites, and participants at trainings and meetings • Reiterate need, intervention, outcomes to ensure mutual understanding • Site monitoring • Continuous program improvement • Using Results/Data and Reporting • Progress reports • Marketing and promotion • CELEBRATE SUCCESS!!!

26. Overview of Prohibited Activities Performance Measures – Outputs and Outcomes • OUTCOMES • Reflects the changes or benefits that occur • Can reflect changes in individuals, organizations, communities, or environment • Address changes in attitudes/beliefs, knowledge/skill, behavior, or conditions • OUTPUTS • Amount of services being provided • People served • Products created • Programs developed

27. Overview of Prohibited Activities Performance Measures – Outcomes Outcome Example: EDUCATION

28. Overview of Prohibited Activities Performance Measures - Alignment Aligning Theory of Change with Performance Measures

29. Overview of Prohibited Activities Performance Measures - Alignment Alignment: Logical flow and connection between the different elements. • Alignment helps you: • Strengthen your theory of change • Articulate you performance measures • Measure more accurately • Cleary report change

30. Overview of Prohibited Activities Performance Measures - Alignment Example: Aligned Output-Outcome with Performance Measure

31. Overview of Prohibited Activities Performance Measures – Impact Evaluation Differences Between Impact Evaluations and Performance Measures

32. Overview of Prohibited Activities Performance Measures- Summary Key Points • Performance measurement is a systematic process of measuring progress (outputs and outcomes) • Performance measurement does not seek to “prove” a theory of change, but can provide snapshots. • Impact evaluation can determine if results occurred because of the intervention. • Strong performance measures align with the theory of change (need, intervention and outcome)

33. Overview of Prohibited Activities Performance Measures- Summary Additional Resources CNCS Priorities and Performance Measures: http://www.nationalservice.gov/resources/npm/home Program-Specific Notices of Funding Opportunities and Application Instructions: http://www.nationalservice.gov

34. Project Director Training Saratoga, NY October 7th – 9th, 2013 Questions?