What will your program achieve? How will you tell? Evidence-based Program Evaluation

1. What will your program achieve? How will you tell? Evidence-based Program Evaluation Updated December 14, 2011 Mary Campbell-Zopf, Ohio Arts Council Mary.Campbell-Zopf@oac.state.oh.us Craig Dreeszen Ph.D., Dreeszen & Associates craig@dreeszen.com

2. Your Instructors Mary Campbell-Zopf Craig Dreeszen is deputy director at the Ohio Arts Council and has been with the agency since 1989. Between 1989 and 2011, Ms. Campbell-Zopf helped secure $12 million for state-level arts, arts education and international programming. She also played a central role in an agency-wide effort to expand the OAC’s International Program through a $1.2 million USDE grant, for which she served as the evaluation directs Dreeszen & Associates, a consulting firm in Northampton Massachusetts. He is a planner, educator, program evaluator, and organizational development consultant. For twelve years he directed the Arts Extension Service at the University of Massachusetts. Since 1986 he has evaluated 30 programs for foundations and state arts agencies and provided strategic planning support to over manager and participated in professional development activities with educators and arts administrators in Chile. In 2006, Campbell-Zopf oversaw development of The Appreciative Journey: A Guide to Developing International Cultural Exchanges. Ms. Campbell-Zopf has an enduring interest in strategic planning and program evaluation, which led to the publishing of the seven-volume series, Focusing the Light: the Art and Practice of Planning in 2009. Ms. Campbell-Zopf has also been active at the state and national levels in arts education, including serving on numerous state advisory committees for content standards, curriculum, learner assessment, as well as on national advisory committees for the U.S. Department of Education, the National Endowment for the Arts, the National Assembly of State Arts Agencies and the Arts Education Partnership. 50 public and nonprofit organizations, mostly community based cultural organizations and the agencies that fund and support them. Dr. Dreeszen is a contributing editor of Fundamentals of Arts Management, 5th Edition, author of the chapters, “Program Evaluation,” “Strategic Planning,” and “Board Development.” He wrote the online courses “Strategic Planning” and “Outcome Based Program Evaluation,” and other arts management books and articles. Craig Dreeszen earned his Ph.D. in regional planning and M.A. in organizational development from the University of Massachusetts Amherst.

3. Our intended learning outcomes • Understand logic and language of evaluation • Define outcomes and indicators • Design feasible program evaluations • Answer funders: “How will you know if your proposed project will succeed?” “What results did your project achieve?” Muse Machine artist Michael Bashaw, Thom Meyer photographer

4. Evaluation Part I Outline • Why evaluate • Link planning and evaluation • Evaluation types • Monitoring activities and measuring outcomes • Logic of evaluation • Language of evaluation • Evaluation questions • Outcomes and indicators

5. Benefits of Evaluation • Greater insight into people's needs and interests • Increased efficiency, economy, and effectiveness • Improved visibility leading to greater public support • Increased funding

6. An Effective Evaluation • Structured on defined evaluation questions and information needs • Logical, defined method of collecting data • Unbiased and culturally-sensitive • Positive and useful process

7. Why Evaluate Programs? • Be accountable • Funders expect you to deliver results as promised • “If your project is funded, how will you know if it succeeds?” • Improve programs • Use evaluation to tell what is working and what needs improvement • Measuring results critical to program planning

8. What will change? What results do you want? What is situation now?

9. Pre-test How confident are you right now that you could answer the funder’s question: “If we fund your project, how will you know if it is successful? • Confident • Somewhat confident • Not very confident • Not at all confident

10. Evaluation in plain English What changes will your program make in the world? = Outcomes What evidence would you accept? = Indicators Did your program make the difference you intended? = Outcome-based program evaluation

11. Planning and Evaluation • Planning determines goals and objectives • Outcome-based evaluation measures achievement of planned objectives

12. Planning and Evaluation Planning and Evaluation Plan objectives  Observe outcomes Just like financial management Budgeting and Accounting Project budget  Report actual revenues & expenses

13. Build evaluation into your plans "Would you tell me which way I ought to go from here?" asked Alice. "That depends a good deal on where you want to get," said the Cat. "I really don't care where" replied Alice. "Then it doesn't much matter which way you go," said the Cat. Lewis Carroll

14. Evaluation Defined Evaluation The process of gathering objective evidence about a program and using that evidence to make judgments that guide decisions. Girl Mask, OAC AIR artist Kate Kern

15. Assessment Defined Learner assessment The process of describing, collecting, recording, scoring, and analyzing information about student knowledge, skills and dispositions against instructional objectives and standards of quality. OAC AIR artist Joshua Brown teaches dance residency

16. This is Not a Pipe by Rene Magritte Don’t mistake evaluation for reality. It is a flawed approximation, so keep your perspective.

17. Program has clear objectives • Determine extent to which objectives are achieved • Note unanticipated outcomes • Program objectives are not clear • Discover outcomes • Clarify objectives for planning Two likely situations

18. Formative • in-progress evaluation for better programming • may be informal • often done internally • may be oral • Summative • final report for accountability • more formal • often an outside evaluator • usually written Two kinds of program evaluations

19. Logic Model

20. If-then Logic

21. Logic Model Elements

22. Observable outcomes -- tangible results • Indicators -- evidence of intangible results • Not long-range or general goals • Exceptions: • Some evaluations are not feasible, given time and $ • “Goal” may be used to describe almost any intention What can be evaluated?

23. Glossary • Outcome: a specific result attributable to your program - a specific benefit to participants (delight audiences) • Activity: what program participants do to achieve outcome (attend concert) • Outputs: immediate products of activities that may not be a specific benefit to participants (present 2 concerts for 400 people) • Indicators: measurable evidence of outcome achievement (positive response on exit poll)

24. Near-equivalent terms

25. Writing outcomes If the goal is: “Connect elders’ learning experiences with heritage through the arts.” What’s wrong with this outcome? “A guest historian will use theatrical techniques in a three-part workshop.”

26. Writing objectives as outcomes Objective framed as activity: “A guest historian will use theatrical techniques in a three-part workshop for elders.” If your draft plan describes activities only, add the intended result of these activities. “We do ____ to achieve ____.” If you don’t do this when planning programs, you must do so later when evaluating outcomes. Objective framed as outcome: “A guest historian will use theatrical techniques in a three-part workshop to help elders illuminate their personal stories through the heritage of their times.”

27. Activity yields outcome Goal “Use the arts to transform learning.” Activity “Poet in residence works with senior center residents to write reflective journals.” Intended outcome: “Residents will improve writing skills through journal writing and poetry.”

28. Indicators are evidence of outcome Outcome: • Participants will encounter the poet in residence and write reflective journals Indicators: • Instructors will report the number of participants who produced journals • Instructor reads journals and observe number that are reflective • May use a rubric to standardize observations

29. Rubrics Rubric for Outcome: Students learn dance step • Not yet - little evidence student can accomplish step • Almost - student exhibits technique but with weakness • Meets standard - student exhibits technique • Exceeds standard - meets standard with enthusiasm

30. Relax • You don’t have to evaluate every program • Start with one • You don’t have to measure every outcome • Start with a few • Reject indicators too expensive or time consuming to measure • Gather data in routine managing of programs

31. Challenging evaluations • If contact with beneficiaries is short term • measure short-term outcomes and outputs • If outcomes will take a long time to achieve • measure progress • If outcomes are intangible -- Joy • use qualitative measures -- sample opinions

32. Output: How many participated? Satisfaction: Did they enjoy the program? Short-term outcome: Did they learn what was intended? Long-term outcome: Did they use new learning? Impact: What impact on lives, on community? Some results more profound than others

33. Proving results Easier to report outcomes than toprove your program caused them • Cause and effect may be difficult to prove • Other factors may have influenced outcomes • You can report a correlation • We ran program and observed this result… Cleveland Public Theatre Y-Haven Project Performance

34. Homework • Take short course, “Shaping Outcomes” • Write at least two observable outcomes you expect to achieve for the program you wish to evaluate • Write at least two indicators for each planned outcome • Post your outcomes for feedback