Outcomes and Indicators

Outcomes and Indicators Chesapeake Bay Funders Network Program Evaluation Training Workshop OMG Center for Collaborative Learning January 9-10, 2008

Building an Evaluation Plan • Theory of Change/Pathway Map • Outcomes • Indicators • Methods • Putting Evaluation to Work for You

What are outcomes?  Outcomes are changes in behavior, skills, knowledge, attitudes, condition or status.  Outcomes are related to the core business of the project, they are realistic and attainable, they are within the project’s sphere of influence, and they are appropriate.  Outcomes are what a project is held accountable for.

Start with Logical and Reasonable Outcomes • What makes an outcome logical and reasonable? • There is a logical connection between our activities and the outcome. • It is reasonable to believe that the outcome can be accomplished within the timeframe that we have established. Based on: • Our previous experience • Context (e.g. current or upcoming political changes) • Resources

Start with Logical and Reasonable Outcomes • What makes an outcome logical and reasonable? • The information we are collecting to track this outcome will help us to know more about how we might improve this project or our organization. • Each outcome specifies: • The group or condition or process targeted for change and • What we hope to change about it

Start with Logical and Reasonable Outcomes The group or condition or process targeted for change Example 1: Farmers Example 2: Water Quality and What we hope to change about it Example 1: Use of Fencing and Buffers to Restrict Cattle Access to Creek Example 2: Improved (e.g. reduced sediment concentrations).

Examples of Outcomes Short Term Farmers learn benefits of fencing and buffers Farmers discuss options with agricultural agents, other colleagues

Examples of Outcomes Long Term Farmers install buffers and fences to restrict cattle access to the creek Sediment and nutrients from cattle in creek decrease allowing creek to meet designated uses

Indicators and Targets

What are Indicators? • Indicators are specific characteristics or changes that represent achievement of an outcome. • Indicators are directly related to the outcome and help define it.

What are Indicators? • Indicators are measurable, can be seen, heard or read, and make sense in relation to the outcome whose achievement they signal. • Indicators are what you measure to determine whether participants have achieved an outcome, or to determine the presence or absence of a condition or event.

Targets • Targets specify the amount or level of outcome attainment that is expected, hoped for, or required. • Targets include the level of change in the indicator and the time by which the desired change must be completed. Example: The percentage of farmers in x county using buffers and fences to keep cattle from the creek will increase from 30% to 60% over the next three years.

Targets: Some words of caution • Set targets that are realistic so they are not over or under ambitious. • If the target statement indicates change in magnitude (i.e., increases or decreases), be sure to specify the initial levels and what is considered a successful change. • Be sure target statements are in sync with meaningful project time frames.

Targets: Some words of caution • Performance targets should be specified in advance (i.e., when deciding to measure outcomes). • Be sure there is buy-in regarding what constitutes a positive outcome – when the project has achieved the target and when it has missed the mark. • When lacking data on past performance, it may be advisable to wait to set targets.

What is the difference between outcomes, indicators, and targets? Outcome Increase participation in activism of local constituents through e-network Indicator Number of new e-activists added to e-network during grant period equals or exceeds 2000 Target

Identifying Indicators: Questions to ask yourself • What does this outcome look like when it occurs? • What would tell us it has happened? • What could we count or track? • Can you observe it? • Does it tell you whether the outcome has been achieved?

Identifying Indicators: Use the ‘I’ll Know it When I See it’ Rule The “I’ll know it when I see it” rule in action -- some examples: I’ll know that I have increased the engagement of volunteer river watchers when I see an increase in the number of calls we receive from river watchers notifying us of concerns warranting attention and when I seean increase in letters sent to local politicians by river watchers raising concerns regarding the river.

Examples of Outcomes and Indicators

Final Thoughts to Consider When Identifying Outcomes and Indicators • There is no correct number of outcomes and no correct number of indicators for each outcome • Be sure to think about when to realistically expect outcomes and when to measure them • Shorter term, Intermediate, Longer Term • Outcomes, especially long-term, can go beyond the project’s formal conclusion

So how do we know if our indicators pass the test? TIMEFRAME Are changes likely in the timeframe that you’ve defined? RESOURCES Do you have staff, money, relationships to monitor over time? PAST EXPERIENCE Do lessons learned by similar programs or efforts indicate this change is likely? Indicators LITMUS TEST PROGRAMMING Does your initiative provide services directly related to the indicator? C0NTEXT Is the indicator appropriate in this context & environment?

Exercise: Outcomes and IndicatorsWhich is an outcome? Which an indicator? • Reduced amount of impervious surface on properties • Redirection of downspouts to flow to lawn or garden • Private Industry Absorbs Some Responsibility for Operations and Maintenance (O&M) for AMD Treatment • Properties were modified for greater storm-water runoff sustainability • Company X agrees to construct an AMD treatment system and to have designated 0&M responsibilities for the system which are developed and agreed to in partnership with Y

Exercise (Small Groups) • Refer to the partially completed Theory of Change for the Fullojunk Watershed Alliance and the description of the Alliance’s work. • In addition to the example outcome, develop at least two short term outcomes and one long term outcome for the Alliance. • Develop indicators for those outcomes. (How will you know the outcome has happened - when you see…?)

Outcomes and Indicators