Measuring and Using Outcomes for Volunteer Programs

Measuring and Using Outcomes for Volunteer Programs 2014 Greater Richmond Association for Volunteer Administration Conference July 10, 2014

Topics • Outcome measurement overview: What, why, how • Identifying outcomes • Identifying your volunteer program’s outcomes • Identifying indicators, data sources, data collection methods • Identifying indicators, data sources & data collection methods for your volunteer program • Using outcome data to learn and improve • Parting advice • Resources Measuring and Using Outcomes for Volunteer Programs

What Measuring and Using Outcomes for Volunteer Programs

INPUTS ACTIVITIES OUTPUTS OUTCOMES • Resources dedicated to or consumed by the program • e.g. • What the program does with inputs to fulfill its mission e.g. • The direct products of program activities • e.g. • Benefits or changes for participants during or after program activities • e.g. • number of classes taught • number of counseling sessions conducted • number of educational materials distributed • number of hours of service delivered • number of participants served • feed and shelter homeless families • provide job training • educate teachers about signs of child abuse • counsel pregnant women • provide mentors for youth • money • staff & staff time • volunteers & volunteer time • facilities • equipment & supplies • new knowledge • increased skills • changed attitudes or values • modified behavior • improved condition • altered status Program elements Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996 Measuring and Using Outcomes for Volunteer Programs

not how many worms the bird feeds its young, but how well the fledgling flies Outcome: Measuring and Using Outcomes for Volunteer Programs

Outcome vs. indicator vs. target • Outcome: Benefit or change for intended beneficiaries or target group e.g., Parents will use more effective discipline methods • Outcome indicator: The specific statistic used to show how fully the program is achieving the outcome e.g., The number and percent of parents who, after completing the program, explain rules and consequences and apply consequences each time a rule is broken • Outcome target: Numerical objective for a program’s level of achievement on an outcome e.g., 75% of parents will explain rules and consequences and apply consequences each time a rule is broken Measuring and Using Outcomes for Volunteer Programs

A note about targets • Don’t set targets for outcome achievement until you have collected 1 or 2 rounds outcome data • Until you know your current level of achievement you won’t know what target is reasonable; you’ll have to guess at a target • Whether you meet, exceed, or fall below the target won’t reflect your effectiveness; it will reflect how well you guessed Measuring and Using Outcomes for Volunteer Programs

Outcome measurement is NOT. . . • Experimental research • Program evaluation • Performance measurement • A replacement for tracking inputs, activities, and outputs Measuring and Using Outcomes for Volunteer Programs

Outcome findings will NOT. . . • Show whether what you are measuring is the right outcome for your program’s activities • Prove statistically that your program caused the outcome • Explain why this level of outcome was achieved • By themselves, tell how to improve the outcome Measuring and Using Outcomes for Volunteer Programs

So… Measuring and Using Outcomes for Volunteer Programs

Why Measuring and Using Outcomes for Volunteer Programs

 Increase effectiveness  Communicate value Why measure outcomes? Measuring and Using Outcomes for Volunteer Programs

Internal uses of program outcome findings • Provide direction for staff • Identify training needs • Improve measurement and/or program • Support long-range & annual planning • Guide budgets & justify resource allocations • Suggest outcome targets • Focus board members’ attention on programmatic issues Measuring and Using Outcomes for Volunteer Programs

Externaluses of program outcome findings • Recruit talented staff and volunteers • Promote the program to potential participants and referral sources • Identify partners for collaboration • Enhance the program’s public image • Retain and increase funding Measuring and Using Outcomes for Volunteer Programs

Benefits of program outcome measurement • Executives of 298 programs agreed or strongly agreed – outcome measurement had helped their program: • Focus staff on shared goals (88%) • Communicate results to stakeholders (88%) • Clarify program purpose (86%) • Identify effective practices (84%) • Compete for resources (83%) • Enhance record keeping (80%) • Improve service delivery (76%) • Nearly all (89%) would recommend outcome measurement to the director of a similar program Source: Agency Experiences with Outcome Measurement. United Way of America, 2000 Measuring and Using Outcomes for Volunteer Programs

How Measuring and Using Outcomes for Volunteer Programs

Outcome measurement: 8 steps 8. Use your findings 7. Improve your system 6. Analyze and report your findings 5. Try out your outcome measurement system 4. Prepare to collect data on your indicators 3. Specify indicators for your outcomes 2. Choose the outcomes you want to measure 1. Get ready Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996 Measuring and Using Outcomes for Volunteer Programs

In good outcome measurement … • Outcomes are appropriate for the program • Indicators provide useful data on outcome achievement • Methods – data sources and data collection procedures – are sound • Tools measure what they’re supposed to and are appropriate for the population • Analysis shows level of achievement for different segments of the target audience and different program characteristics Measuring and Using Outcomes for Volunteer Programs

“Outcome” The specific results(s) sought for a specific audience (clients, participants, visitors, patrons, volunteers, etc.) Measuring and Using Outcomes for Volunteer Programs

Identifying outcomes First question: Measuring and Using Outcomes for Volunteer Programs

Volunteer programs: Results for two audiences • Results for clients/ visitors/patrons • e.g.: • Clients receive desired/ needed information • Patrons recommend the setting to others Volunteers’ activities • Results for volunteers • e.g.: • Volunteers provide accurate information • Volunteers make a long-term commitment Volunteer program administration activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 • Results for clients/ visitors/patrons • e.g.: • Clients receive desired/ needed information • Patrons recommend the setting to others Worksheet A: Activities of your volunteers Volunteers’ activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 • Results for clients/ visitors/patrons • e.g.: • Clients receive desired/ needed information • Patrons recommend the setting to others Worksheet B Section II: Outcomes you want your volunteers to create Volunteers’ activities Measuring and Using Outcomes for Volunteer Programs

THEN … this change THEN … this change THEN … this change IF … these activities Activities  Outcomes: An “If…then” chain Measuring and Using Outcomes for Volunteer Programs

THEN … this change in condition/status THEN … this change in behavior THEN … this change in knowledge IF … these activities A simple “If…then” chain Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 • Results for clients/ visitors/patrons • e.g.: • Clients receive desired/ needed information • Patrons recommend the setting to others Volunteers’ activities Worksheet C: If … then chain linking volunteers’ activities and outcomes for target audience Measuring and Using Outcomes for Volunteer Programs

Testing your “If … then” chain • Are the outcomes really outcomes? Do they describe changes in or benefits for clients/ participants/beneficiaries, rather than something about the program? • Is the longest-term outcome meaningful for program clients/participants/beneficiaries? Will the change or benefit make a real and lasting difference for them? • Is the logic logical? Are the activities sufficient to produce the intended outcomes for these clients/participants/beneficiaries? Measuring and Using Outcomes for Volunteer Programs

At-risk teens graduate from high school. At-risk teens meet district grade requirements. At-risk teens complete homework regularly. At-risk teens know homework is important. Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, may help teens with homework. Are the activities sufficient to achieve the outcomes? Outcomes Activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 1 Worksheet D: Testing your If … then chain Outcome B Outcome A Activity 2 Activities 3 & 5 Outcome C Activities 1 & 4 Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 2 Worksheet E: Impact you want to have on your volunteers • Results for volunteers • e.g.: • Volunteers provide accurate information • Volunteers make a long-term commitment Volunteer program administration activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 2 • Results for volunteers • e.g.: • Volunteers provide accurate information • Volunteers make a long-term commitment Worksheet F: If … then chain linking volunteer program administration activities and outcomes for volunteers Volunteer program administration activities Measuring and Using Outcomes for Volunteer Programs

Identifying your volunteer program’s outcomes Part 2 Worksheet G: Testing your If … then chain Outcome B Outcome A Activity 2 Activities 3 & 5 Outcome C Activities 1 & 4 Measuring and Using Outcomes for Volunteer Programs

OUTPUTS Identifying your volunteer program’s outcomes ACTIVITIES INPUTS • Requires time, thought, discussion • Benefits from input by volunteers, clients/visitors/patrons, board members, and engaged donors as well as staff • Evolves over time • Provides content for your program’s “logic model” • Is worth the effort OUTCOMES Measuring and Using Outcomes for Volunteer Programs

Programs report that defining outcomes and creating logic models … • Stimulates discussion of the program’s purpose • Creates a shared understanding of how activities are intended to link to results • Often prompts adjustments in activities and intended results • Helps communicate the program’s intent to stakeholders • Makes sure useful data is collected – and unnecessary data is not • Guides review of data to identify improvement opportunities • Often leads to improved outcome performance because now everyone understands the results the program is aiming for Measuring and Using Outcomes for Volunteer Programs

“Indicator” • Specific statistic used to show how fully the outcome is being achieved • Often the number and percentage of clients/patrons/ volunteers who, e.g.: • attain specific knowledge • demonstrate a needed skill • display a certain behavior • achieve a desired status Measuring and Using Outcomes for Volunteer Programs

Examples of indicators • Number and percentage of volunteers in the rotating gallery who can answer 25 FAQs about the current exhibit correctly • Number and percentage of activity volunteers in the senior center who use techniques for engaging withdrawn seniors • Number and percentage of fund-raising volunteers who renew their commitment for 1 year of service • Number and percentage of children in the activity program who read at least 5 books with their volunteer reading buddy • Number and percentage of patrons in the volunteer-staffed gift shop who make or add to a purchase Measuring and Using Outcomes for Volunteer Programs

Identifying possible indicators Questions to ask to generate options: • How will we know if the outcome is achieved? What will we see? What can we count or measure? If you’re stuck, ask: • How would we know if the intended result is not achieved? What would we see? • If we were successful in one instance but not another, how would we know? What would we see in one instance but not the other? Measuring and Using Outcomes for Volunteer Programs

Testing a possible indicator • Does it capture the central point of the outcome? (e.g., if the central point is behavior, it tracks behavior – not knowledge) • Is it specific enough that everyone interprets it the same way? • Is it unlikely to be influenced by factors others than the outcome? • Do key stakeholders accept it as a measure of the outcome? Measuring and Using Outcomes for Volunteer Programs

Possible sources of program outcome data • Case/client records • Documents • Data files • The volunteers/clients/visitors/patrons themselves • Others who know the volunteers/clients/visitors/patrons • Other individuals • Physical environment Measuring and Using Outcomes for Volunteer Programs

Possible methods of collecting program outcome data • Record review • Document review • Data file review • Questionnaire • Interview • Observation • Testing • Mechanical measurement Measuring and Using Outcomes for Volunteer Programs

Identifying indicators, data sources & data collection methods for your volunteer program Worksheet H: Indicators, data sources, data collection methods for your outcomes Measuring and Using Outcomes for Volunteer Programs

Using outcome data to learn and improve • Calculate overall outcomes — # and % of target audience achieving the outcome e.g.: Total number served 1,000 Number achieving outcome 535 Percent achieving outcome 53.5% Measuring and Using Outcomes for Volunteer Programs

Using outcome data to learn and improve • Calculate overall outcomes • Break out results by: • level of outcome • relevant characteristics of target audience • relevant program characteristics • combinations to see where you are having more and less success Measuring and Using Outcomes for Volunteer Programs

At-risk teens graduate from high school. 60% 85% Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework. Outcomes by level of outcome At-risk teens achieve passing grades. At-risk teens meet district attendance requirements. At-risk teens earn better grades. At-risk teens attend school regularly. At-risk teens complete homework regularly. Measuring and Using Outcomes for Volunteer Programs

At-risk teens graduate from high school. Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework. Outcomes by client characteristic:Parent attended teacher conference At-risk teens achieve passing grades. At-risk teens meet district attendance requirements. Yes 75% At-risk teens earn better grades. No 45% At-risk teens attend school regularly. At-risk teens complete homework regularly. Measuring and Using Outcomes for Volunteer Programs

Measuring and Using Outcomes for Volunteer Programs