DIAC Session 3, November 18 2010 Using evidence & Managing stakeholders

1. DIAC Session 3, November 18 2010Using evidence & Managing stakeholders Professor Adam Graycar Centre for Policy Innovation

2. Evidence used • To assess the nature and extent of the problem • To assist in reforming policy • To assist in developing new policy • Policy happens faster if the knowledge base is good and if lessons from previous activities can be brought to bear adam.graycar@anu.edu.au

3. Evidence for • What is the situation/ problem • What should be done • Is it working? Evaluating effectiveness • What might go wrong? / unintended consequences adam.graycar@anu.edu.au

4. Evidence • is information that affects the existing beliefs of key people about the problem and how it might be solved • Information • comprises data that has “meaning” - can help sort things into logical or empirical categories • Data • are facts about the world - stats, observations adam.graycar@anu.edu.au

5. Data measures or describes • Trends • Preferences • Finance • Performance • Impacts • Evaluation • Benchmarks • Forecasts • etc etc adam.graycar@anu.edu.au

6. Where do you find data? • ABS • Treasury • AIHW • ABARE • AIC • Budget papers • Annual reports/ performance reports • CIA • OECD/ UNDP/ ISSA/ WHO • World Bank / IMF • Cochrane/ Campbell • etc etc etc etc adam.graycar@anu.edu.au

7. Collecting data • Surveys • Observations • Interviews & Focus groups • Modelling • Scenario development • etc etc adam.graycar@anu.edu.au

8. Think before you collect • Review the available literature • Survey best practice • Use analogies • Talk to opponents • Construct alternatives • Start comprehensive, end up focussed • (Bardach) adam.graycar@anu.edu.au

9. Use existing research - commission new research? • Qualitative/ Quantitative • Cross-sectional/ Longitudinal • Prevalence /Incidence • When research contradicts itself? adam.graycar@anu.edu.au

10. Key evidence resources • Research - both domestic & international • Statistics - both domestic & international • Policy evaluations • Expert knowledge • Economic modelling & forecasting • Cochrane Campbell adam.graycar@anu.edu.au

11. A HIERARCHY OF EVIDENCE OF IMPACT(UK CABINET OFFICE) • Systematic review – synthesis of results from various studies (preferably randomised trials) • Randomised policy trials – population allocated randomly to groups • Quasi-experimental study – similar populations compared • Pre-post study – results compared before and after intervention (Higher ones more reliable; lower ones much more frequently used in Australia) adam.graycar@anu.edu.au

12. When evidence is equivocal? • The political process sorts itself out • The analyst shapes the alternatives adam.graycar@anu.edu.au

13. Evaluation • 3 questions • Any good? • Better than what we had before • Worth it? adam.graycar@anu.edu.au

14. Process evaluation • Impact evaluation adam.graycar@anu.edu.au

15. EVALUATION TYPES • Process evaluation Was the program delivered to the intended people? Were they satisfied? Was the administration efficient? Were the costs acceptable? • Impact evaluation Was the program effective? Did it achieve the intended outcomes such as poverty alleviation, savings, spread of lifetime earnings? Did it have unintended effects such as on employment, earnings, savings? adam.graycar@anu.edu.au

16. PROCESS EVALUATION • It may include • whether the right people got the right money/services and that no others did; • that the services were provided in a timely and convenient way • that people considered the programs were beneficial, helping when needed, being reliable etc • that the administrative costs were low, and total costs in line with estimates adam.graycar@anu.edu.au

17. IMPACT EVALUATION • First round impact • Static picture of impact on clients circumstances (eg income distribution, immediate education/training access, employment) • May include surveys of the circumstances of beneficiaries (lifestyle, location, housing, health etc) • Second round impact • Dynamic picture of people moving in and out of education, work and welfare, longer-term impact; time series • Post-hoc examination of effects of incentives to work, to gain skills, to save; surveys of behaviour adam.graycar@anu.edu.au

18. IMPACT EVALUATION • Third round review • What is the counter-factual? What might have happened anyway, or under different scenarios and policies? • Randomised trials, dynamic modeling adam.graycar@anu.edu.au

19. Inquiries • Task Forces • Reviews adam.graycar@anu.edu.au

20. Commissions of Inquiry • Considerable independence from government, reports published, public consultations; may have secretariat of seconded officials to do research etc • Government terms of reference and appointment of Inquiry head or team adam.graycar@anu.edu.au

21. Task Forces • Often a mix of government and external experts • May have a ‘steering committee’ of senior government officials • Limited independence, limited public consultations, though task force signs off report which is usually published adam.graycar@anu.edu.au

22. Reviews • May involve external reviewer, or advisory group, but usually closely controlled by government • Reports usually published adam.graycar@anu.edu.au

23. examples • Inquiries • Poverty Inquiry (early 1970s) • National Superannuation Inquiry (mid 1970s) • National Compensation Inquiry (mid 1970s) • Royal Commission on Australian Government Administration (mid 1970s) • National Health and Hospitals Reform Commission (2007-08) adam.graycar@anu.edu.au

24. examples • Task Forces • Reference Group on Welfare Reform (2000) • Task Force on the Delivery of Health and Aged Care Services (2004-05) • Review of the Child Support Scheme (2005) • Reviews • Income Security Review (1970s) • Social Security Review (1980s) • Review of the Tax-Transfer System (2008-09) • Review of Australian Gov’t Administration (2009-10) adam.graycar@anu.edu.au

25. Who are the stakeholders? Interests? • Clients/customers • Those subject to regulation • Provider organisations • States and local government • Advocates • Broader public adam.graycar@anu.edu.au

26. Stakeholders/ interests? • Clients/customers • Quality and effectiveness of services • Levels of subsidies • Limiting their obligations • Those subject to regulation • Minimum red tape, clarity, consistency • Minimum interference in their business, impact on profitability • Provider organisations • Minimum red tape, policy continuity • Long-term contracts/agreements • For-profits focus on profitability; not-for-profits more on clients/customers adam.graycar@anu.edu.au

27. Stakeholders/ interests? • States and local government • Recognition of their role and expertise, sovereignty • Maximising revenues: central agencies wanting budget flexibility; line agencies liking conditions that guarantee they get the money • Broader public • Limited taxes • Maximum effectiveness adam.graycar@anu.edu.au

28. Stakeholder processes • Consultation • Collaboration / building bridges • Co-production • Problem identification, evaluation, • Development of policy options • Post-decision processes / implementation adam.graycar@anu.edu.au

29. Examples of stakeholder relations • Consultation • Includes feedback from complaints systems, service charters, ministerial correspondence • As well as special forums, discussion paper responses etc • Collaboration • Joint evaluations, reviews • Program arrangements allowing providers and/or clients discretion within agreed objectives/guidelines adam.graycar@anu.edu.au

30. Examples cont’d • Co-production • Program requires contribution – time, effort, not just money - from clients to be effective • Similarly, policy cannot be determined without substantial client involvement, including evaluation adam.graycar@anu.edu.au

31. Processes of engagement • Evaluation, problem identification, analysis • Ongoing consultation with client and provider organisations • Systemic feedback from program administration, including through use of new technology • Public inquiries (range of options) • Joint reviews • Development of policy options • Informed judgments of likely views • Market research • Targeted consultations, often confidential • Use of internet communications with public, or selected external groups • Green Papers, other public documents adam.graycar@anu.edu.au

32. Processes of engagement (cont) • Post-decision processes • Targeted consultations or negotiations • Targeted communications • Post-announcement, implementation • Consultations/negotiations on details • Implementation of communications strategy • Managing responses to parliamentary discussions of legislation etc • Establishment of ongoing program monitoring with stakeholders adam.graycar@anu.edu.au

33. Issues in stakeholder engagement • Understanding interests and conflicts of interests • Managing confidentiality • Ensuring accountability when responsibility is shared • Respective roles of ministers, advisers and public service • Legitimate and improper communications management • Risks of delays, compromises not in the public interest etc vs achieving worthwhile, sustainable results adam.graycar@anu.edu.au

34. Stakeholder management adam.graycar@anu.edu.au