An Approach to Evaluation for Smaller and Grassroots Organizations

Level Best An Approach to Evaluationfor Smaller and Grassroots Organizations By Marcia Festen and Marianne Philbin With special thanks to the Pratt Richards Group for additional slides and content

Learning Objectives • Become more familiar with basic principles of evaluation • Learn an approach that will make ongoing evaluation easier

Our Discussion Today • Based on the premises, principles and steps outlined in Level Best • 5-Step Framework • Worksheet

The Challenge “It is easy to tell if you are succeeding in business— you make money. In philanthropy, measuring performance can be fiendishly tricky and take a lot longer.” Warren Buffet “The New Powers In Giving” The Economist, June 2006

Why Evaluate Your Program? • To learn, plan and improve. With limited resources, you need to be as strategic as possible • To better articulate the progress you’re making toward your goals

But First… Pick a card

Your Experience • What’s been the best and worst you’ve experienced relative to undertaking evaluation? • Your assumptions, anxieties, realities?

A Few Basic Principles Evaluation becomes less daunting when you know that…

It’s Not Research • Monitoring – measurement, on an ongoing and regular basis, of program implementation, or results of a service. • Research– the systematic process of collecting data in a controlled environment in order to prove or disprove a hypothesis.

Definition of Evaluation Evaluation is the systematic process of asking questions, and then collecting information to help answer those questions, in order to improve the work of your organization, and often to tell the story of change.

Shift your thinking …from ‘prove’ to ‘improve” …from ‘judge’ to ‘learn’

…from ‘after’ to ‘before’ • Rather than thinking of evaluation as the test that follows the work, begin to think of evaluation as the measures you put in place beforehand to help you run your programs. • How will you evaluate the work you’re planning for this year, or next year?

Many methods, much jargon Process evaluation, outcome evaluation, summative, formative, participatory, Logic Model, quasi-experimental design etc.

Just start by asking: • What do we want to know? which is generally: ARE WE MAKING PROGRESS TOWARDS OUR GOALS?

Quick Exercise • “I wish I knew whether…” _______________________ .

Level Best Evaluation Steps 1. Planning…… why/what you will evaluate • Asking……… one-twokey questions you want answered • Tracking…… the activities that you conduct, the signs that you’re making progress towards your goals • Learning…. from what you track and what it tells you • Using…. the insight you gain to shape your next program

Limit it • Rolling evaluation: our term for an evaluation process wherein smaller organizations choose one or twoquestions to ask, or areas per year to evaluate, and build evaluation learning year by year, over time.

For Example “Although we are running 3 programs as part of our Leadership Development project, this year we will focus evaluation on our Parent Training Program. “The purpose of this particular evaluation is to help us determine whether or not to invest more in 2007 in this particular program.”

Step 1: Planning • Identify a project or area you’d like to evaluate. It can be…. a priority, an idea, a particular program, a strategy within a program, an aspect of your process… Worksheet

Avoid the BIGGEST Mistake Know what you want to evaluate? First step: be absolutely clear on its GOAL. You cannot evaluate something for which no goal has been set !

You can’t just stroll and gather… • It is a common misperception to assume that there is something simply “there” that will become apparent if you “point the camera” and shoot, or go looking in a general way.

How the Goal Shapes the Evaluation Example: “The Kids Will Keep Their Rooms Clean”

For Every Program * GOAL or DESIRED CHANGE: * Rationale: * SIGNS OF SUCCESS – We’ll agree we’ve made progress towards this goal IF: * STEPS to get us there:

Worksheet • Fill in as we go • Take a moment now to work together to fill in Box B, C, D • (for now, skip A !)

SO---Remind Everyone of the PROJECT’S GOAL What is the desired overall outcome for this project? Write down the “goal statement” in Box C Worksheet

Why evaluate this now? • Box D. Write down one purpose you could imagine having for the particular evaluation you might conduct. Worksheet

Must-Have: #1 Agreement as to the CONDITIONthat your organization or program is addressing (is your youth theater program trying to change kids, or change local boring theater?). Clarity as to the CHANGE you hope to make. The question you ask will change depending upon the condition you see yourself addressing. Worksheet

Identifying the “condition” your program addresses For example…

Must-have: #2 A clear GOAL STATEMENT for any program or area you want to evaluate; goal statement typically refers to the effect you want to have on the condition.

Step 2: Asking • Once you determine the purpose of the evaluation, brainstorm with your team three or four questions you’d love to be able to answer better. • Select one or two to explore in this evaluation. Put the others on a “calendar” for next year or the following year.

Types of Evaluation Questions • In Process Evaluation the guiding questions focus on the quality of a program’s components or implementation. • In Outcome Evaluation the guiding questions focus on the extent to which a program is achieving its desired results.

“You” You do work. When you evaluate how well you do what you do, it’s called a PROCESS EVALUATION.

“Them” Your work has results. It influences what people do or believe. When you evaluate what they do, it’s called an OUTCOME EVALUTION.

Impact Lots of work produces multiple outcomes over time. This equals IMPACT

“Theory of Change” The “IF” part of the statement often refers to what your organization will do (process) the “THEN” refers to what you hope will happen as a result (outcome). In some circles, this is referred to as your “Theory of Change.”

So.. are we? / did they? • IF we offer reading tutors, THEN more people will be able to read. • IF we educate teens effectively about sexually transmitted diseases, THEN fewer will engage in unsafe practices.

Theory of Change IF we…. THEN… Which will lead to… Community or Social Change Strategies Immediate Outcomes Intermediate Outcomes Intended IMPACT (long-term)

What’s Most Important not whether you’re doing process or outcome evaluation… simply: What do you want to know?

All flows from The Question • You must ask a specific question: what you track, consider and learn is dependent on the question you ask.

You Can Ask/Evaluate Anything “Are teachers benefitting from the training we offer them? (How do we know?)” “Do visitors like our exhibits? (How do we know?)” “Are we serving girls as well as we serve boys in this program? (How do we know?)”

But you must ask something! • Evaluation is NOT about collecting massive amounts of information, and then attempting to sort through it later to see “what it says”

Step 3: Tracking • List two or three things that if they occurred, would be signs of progress, examples that might show you’re * making progress towards the desired outcome for the project or * that provide information that would help you answer the question you’ve posed.

Indicators What information would tell you if you’re making progress toward your desired outcome? What would you track to help you answer the evaluation question you posed? That’s an “indicator”

Program Example • Desired Outcome: Our theater’s main stage audience engagement increases • How We’ll Know We’re Making Progress/ #1: Mainstage attendance as a percent of total seat capacity will increase to 65% for the 2010-11 season. • How We’ll Know We’re Making Progress/ #2: 90% of mainstage audience members completing the end of season survey will be “very” or “somewhat satisfied” with the season.

Organizational Example • Desired Outcome: Board engagement improves • How we'll know we're making progress / #1: 80% of Board members will contribute $500 or more in FY2011. • How we'll know we're making progress / #2: Each quarterly Board meeting will have 75% attendance or better in FY2011.

Golden Rule In evaluation we want to measure what we can control. So…

Measure What Is in YOUR Control… Not everything that you want to achieve in your program will be measurable.

Track What’s Closest to What YOU do The goal of our tutoring program is to help students… --someday get jobs --graduate from high school -- better homework completion rate -- attend homework helper program -- be referred to this program by teacher Keep drilling down, ask “what needs to happen before that can happen?” You can’t track it all.

S.M.A.R.T. Indicators • Specific • Measurable • Attainable • Relevant • Time bound

Which Indicators to Track • Refer back to the CONDITION you’re hoping to change or address in the first place

An Approach to Evaluation for Smaller and Grassroots Organizations