Getting started with evaluation

1. Getting started with evaluation

2. QUIZ #1 Test your evaluation IQ! On the next slides, think about what evaluation is…then, compare your ideas to a common definition (slide #4).

3. What IS evaluation? (choose one answer) • It is a survey to collect information from participants • It is an end-of-session questionnaire • It is a success story to document our value • It is statistics and data analysis (that I don’t like to do!) • A systematic process of collecting information for a specific purpose and use.

4. While many think of evaluation as synonymous with data collection (a survey or an end-of-session questionnaire) or reporting a success story, it actually is much broader than any one of those things. The first four answers on the quiz are part of what make up evaluation…but evaluation includes more than any one of these things. The correct answer is E.

5. Evaluation is… the systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding. (Patton, 2008:39) This definition of evaluation comes from Michael Patton in his book, Utilization-Focused Evaluation. Notice the phrases in color as key concepts about what evaluation is and why we do it.

6. In other words, “Don’t accept your dog’s admiration as conclusive proof that you are wonderful.” Ann Landers

7. Remember, evaluation actually is part of our every day lives… In fact, we engage in “informal” evaluation all the time. We try a new restaurant and decide whether to go back again. We examine the meat in the meat case at the grocery store and decide which cut, if any, to buy. We watch our program participants to see if they appear to be engaged and learning. We ask our volunteers how things are going. However, sometimes we need to be more deliberate and systematic. To have information that is seen as credible and trustworthy by others, we engage in what we characterize as “formal” evaluation. We commit time and resources in a deliberate process of systematic data collection to have credible information for making sound decisions. See the Program Development and Evaluation booklet, Developing a concept of Extension Program Evaluation; link provided on web site

8. Informal Formal • Casual • Spontaneous • Information is quickly collected • Sometimes insufficient • Credibility of information may be questioned • Deliberate, thoughtful • Takes time • Information is systematically collected • Criteria are explicit • More credible information

9. Judging Learning Images of Evaluation Evaluation is often thought of in terms of judgment . This can provoke fear and anxiety. However, evaluation also helps us learn. This might be the most important and useful way to think about evaluation.

10. What is your image of evaluation? Do you see it as… • judgment? • hard and difficult? • someone else’s responsibility, not yours? • something to fear and loathe? • a waste of time? • Or, do you see the potential of evaluation to foster and enhance learning? Spend a few minutes reflecting on what your image of evaluation is.

11. Quality evaluation Like most professions, evaluation has a set of professional standards that guide ethical and proper practice. These four Evaluation Standards are: • Utility: How useful is the evaluation to you and your audiences? i.e., Will it serve the information needs of the intended users? • Feasibility: How practical is your evaluation? i.e., Will it not disrupt program operations; will it be worth the costs? • Propriety: How appropriate is your evaluation for those involved? i.e., Does it respect people, attend to human subjects protection, is it ethical and legal ? • Accuracy: How accurate is your evaluation and the information it conveys? i.e., do the methods and procedures revel technically adequate information?

12. Use the evaluation standards as guidelines to help ensure high quality evaluation. Quick Tips 9 – Ways to Improve the Quality of Your Program Evaluations will help. Links to the Quick Tips are found on this web page: http://www.uwex.edu/ces/pdande/resources/

13. Get clear on language Quantitative Formative Summative ImpactOutcomes Process AccountabilityOutputs Indicators Measurement Data Inputs There are a lot of technical terms and jargon in evaluation. Be sure when you are working with others that you mean the same thing with the words you use. We often talk past each other. See the Glossary (pdf listed on web site) for a list of definitions of common evaluation terms. Qualitative

14. WIIFM - What’s in it for me? By doing quality evaluation, • Your bragging rights are legitimized. • You can feel sure that you are making a difference and using public dollars well. • You will have credible information to share with my committees and participants. • Your outcomes will be stronger because measuring results produces results!

15. Continued…By doing quality evaluation, • You will know what needs to be done to make my programming stronger. • You can demonstrate your scholarship. • You will increase understanding of your programming purpose and direction. • You will achieve your promotion goals. What else would you add to this list?

16. Make Evaluation Meaningful • Make evaluation a part of programming – not an add-on or something that is required. • Build evaluation in upfront in the planning process • Use your resources wisely – don’t expect to formally evaluate everything. • Use evaluation to “learn”.

17. And, finally… Practice– practice – practice We learn evaluation by doing it!

18. Reflection time What is one thing you learned (or had reinforced) from going through this presentation that you hope not to forget?