Making a Difference Through Program Evaluation

1. Making a DifferenceThrough Program Evaluation Michael Quinn Patton September 21, 2010 Program Evaluation Quality Assurance Summit

2. LBJ

3. Globalization of the evaluation community – of which you are all a part.

4. Burkina Faso

5. Evaluation as a transdiscipline and a profession… with quality assurance Standards and professional development opportunities.

6. Standards – Quality Assurance • Utility – ensure relevance & use • Feasibility – realistic, prudent, diplomatic & frugal • Propriety – ethical, legal, respectful • Accuracy – technically adequate to determine merit or worth For the full list of Standards: www.wmich.edu/evalctr/checklists/standardschecklist.htm

7. Meta-evaluation Evaluating evaluations for ongoing learning, improvement and quality assurance

8. Quality AssuranceEnsuring Value through Evaluation E-VALU-ate

9. Quality Assurance Quality control and/or Quality enhancement

10. Personal Factor: Intended Use by Intended Users

11. Critical success factors: There are five key variables that are absolutely critical in evaluation use. They are, in order of importance: • People • People • People • People • PEOPLE

12. Big news Evaluators are people too

13. Basic premise Value and quality assurance in Evaluation comes through EVALUATORS

14. Essential skills Add value by being good at what you do and knowing your role and playing it well

15. Know thyselfConnais-toi toi-même Greek: γνῶθι σεαυτόν Latin: nosce te ipsum

16. Essential Competencies for Program Evaluators Jean King, a recipient of the American Evaluation Association’s prestigious Alva and Gunnar Myrdal Practice Award, has worked for a number of years with colleagues and students conducting research on and developing a framework for Essential Competencies for Program Evaluators (Ghere et al. 2006;King et al., 2001). • The final product is a taxonomy of essential program evaluator competencies organized into six primary categories.

17. Essential Competencies 1. Professional Practice: knowing and observing professional norms and values, including evaluation standards and principles. 2. Systematic Inquiry: expertise in the technical aspects of evaluations, such as design, measurement, data analysis, interpretation, and sharing results. 3. Situational Analysis: understanding and attending to the contextual and political issues of an evaluation, including determining evaluabiity, addressing conflicts, and attending to issues of evaluation use.

18. Essential Competencies 4. Project Management: the nuts and bolts of managing an evaluation from beginning to end, including negotiating contracts, budgeting, identifying and coordinating needed resources, and conducting the evaluation in a timely manner. 5. Reflective Practice: an awareness of one’s program evaluation expertise as well as the needs for professional growth.

19. Essential Competencies 6. Interpersonal Competence: the people skills needed to work with diverse groups of stakeholders to conduct program evaluations, including written and oral communication, negotiation, and cross-cultural skills.

20. Essential skills Add value by being good at what you do and knowing your role and playing it well

21. The Nature of Expertise and the role of expert

22. What’s it take to achieve world class expertise? 10,000 hours

23. Identities we take on • Evaluator • Methodologist • Researcher • Auditor • Inspector • Learning facilitator • Judge These roles are necessary, important and useful, but there’s another role that offers value: The jester

25. SPEAKING TRUTH TO POWER

26. Jesting is Serious Business because Speaking TRUTH to Power is serious business

27. In the beginning…

29. Loading... Loading...

30. to achieve intended use by intended users: Value-added Challenge: Matching the evaluation process and design to the nature of the situation to achieve intended use by intended users: Contingency-based Evaluation

31. Evaluation’s General, Generic Value Value of a Particular Evaluation and Specific Approach

32. Mrs. McCave and her 23 Daves

33. Not just experience but Focused Practice and REFLECTION on experience to achieve situational responsiveness

34. Story about ongoinglearning

36. Executing Doing what we know how to do and know we should do

37. The Fine Art of Selling • Successful Selling Methods • What Makes a Great Salesperson? • The Secret to Selling • Increasing Sales Now • Sales Tips from Great Salesmen • Topping the Sales Charts • Breakthrough Strategies For Selling to Difficult People • Expert Hands-On Sales Techniques • Improving Sales Performance - Where to Begin? • Sales Techniques Revealed

38. Lessons from Sales Experts • Believe in your product • Know your product • Connect your product to what your customer needs: listen communicate connect

39. Elevator speech test Can you say clearly, in 30 seconds what your product is, why you believe in, and say it in a way that connects with the person you’re talking to? Elevator test

40. Believe in your product

41. Bob Stake, “Beyond Neutrality: What Evaluators Care About”

42. Beyond Neutrality: What Evaluators Care About 1. We often care about the thing being evaluated. 2. We, as evaluation professionals, care about evaluation. 3. We advocate rationality. 4. We care to be heard. We are troubled if our studies are not used. 5. We are distressed by underprivilege. We see gaps among privileged patrons and managers and staff and underprivileged participants and communities. 6. We are advocates of a democratic society. SOURCE: Robert Stake (2004:103–107).

43. Know your product

44. Value-added Challenge: Matching the evaluation process and design to the nature of the situation to achieve intended use by intended users Contingency-based Evaluation

45. Connect your product to what your customer needs: listen communicate connect

48. PROCESS USE

49. Three Cups of Tea Baltistani proverb: First cup you share, you are a stranger. Second cup, you are an honored guest. Third cup, you are in relationship.

50. Enhancing Quality in Evaluation • Believe in evaluation • Know evaluation (and yourself as evaluator) • Connect to your primary intended users: listen communicate connect