Bronte Koop, Principal Consultant, Strategy and Evaluation

1. Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector Bronte Koop, Principal Consultant, Strategy and Evaluation John Collins, Manager, Program Evaluation

2. National Partnerships Literacy and Numeracy Teacher Quality Low SES communities COAG Reform Agenda: Human Capital Development

3. Not getting what you expected from an evaluation

4. What do we mean by Snake Oil • Staff as passive recipients • Evaluation questions not addressed • Use lots of words • Recommendations to continue the program and evaluate it again • Lack of evidence based recommendations for improvement • No discussion of data or design limitations • No discussion of alternative explanations for program effect • Evaluators reluctant to address client concerns about clarity or focus

6. Useful information to guide decision making

7. So what’s the Good Oil • Staff critical participants • Provides answers to specific questions • Balances qualitative with quantitative data • Reports are just sufficient in length • Conclusions follow from the evidence • Has logical recommendations • Data or design limitations are discussed • Alternative explanations for program effect are discussed

8. How do we get there? teamwork commitment learning proactivity

9. Attitudes and values What are we trying to change? Knowledge and skills Organisational behaviour

10. Overcoming your fears is the first step

11. Principles for evidence based program management • Demand evidence • Examine the logic • Treat the program as an unfinished prototype • Embrace an attitude of wisdom • Jeffrey Pfeffer and Robert Sutton on evidence based management

12. CREATINGCHANGE Where should be put our efforts? • Executive leadership • Consulting Services • Resources • Professional development • Policy and operations • Networks • Strategic Oversight

13. Locate near the top Recruit the knowledge, skills and motivation Build credibility Provide an ‘evaluation perspective’ Consulting Services

14. Resources If you build it, they will come

15. What have others done?

16. Accessible Resources

17. Practical Resources

18. Practical Resources Provide a starting point

19. Project Management

20. Evaluation & planning

21. Professional Development • Time to stretch your thinking • Induction • Customized training to needs

22. Policy and Operations Think outside the square Procurement: Expression of Interest Two part contracts: plan + do Executive summaries Interviews Hard questions

23. Network, network, network

24. Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

25. Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

26. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . .

27. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate.

28. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate. DRAFT

30. Represents a major cultural change.

31. Represents a major cultural change • From an ‘accountability’ model to one centred on the corporate goal.

32. Represents a major cultural change • From an ‘accountability’ model to one centred on the corporate goal. A tough call

33. How?

34. A marketing approach

35. A marketing approach Product life-cycle

36. A marketing approach Product life-cycle

37. A marketing approach Product life-cycle Customer

38. A marketing approach Product life-cycle Customer Needs and wants

39. A marketing approach Product life-cycle Customer Needs and wants

40. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

41. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

42. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

43. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

44. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity

45. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity

46. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity Competition

47. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity Competition

48. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity Competition 4 Ps – Profit Product Place Promotion

49. Priorities? To continue development & refinement of our ‘system’ To address changing thefocus of our evaluation culture To move from ‘good oil’ to ‘best value’ oil

50. Disclaimer The views and interpretationsexpressed in this presentation are those of the authors anddo not necessarily reflect those of the Department of Education and Children's Services or the Government of South Australia