1 / 17

David Shand dshand@worldbank

Monitoring and Evaluation in New Zealand Paper for NDRC Monitoring and Evaluation Conference Beijing, 25-26 October, 2006. David Shand dshand@worldbank.org. The New Zealand Context. A small country – 4.2 million people, 15 percent indigenous population

nika
Download Presentation

David Shand dshand@worldbank

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Monitoring and Evaluation in New ZealandPaper for NDRC Monitoring and Evaluation Conference Beijing, 25-26 October, 2006 David Shand dshand@worldbank.org

  2. The New Zealand Context • A small country – 4.2 million people, 15 percent indigenous population • Middle level income – 28th in world, GDP per head of US$25,000 • “Westminster” form of government, reflecting British colonial heritage. Strong executive government but role of Parliament has increased following change in electoral system • Tradition of a strong central government and a progressive social welfare tradition • Key objectives set by the current government • - to strengthen national identity and uphold the principles of • the Treaty of Waitangi; • - to grow an inclusive, innovative economy for the benefit of all; • - to maintain trust in government and provide strong social • services; • - to improve New Zealanders’ skills; • - to reduce inequalities in health, education, employment and • housing; • - to protect and enhance the environment

  3. The Development of Monitoring and Evaluation • Two major strands or initiating factors • - Public Financial Management (PFM) reforms • - Social science research emphasis • Program analysis and evaluation with the introduction of Planning Programming Budgeting system (PPBS) from late 1970 • Indicative National Planning system from 1969 to about 1980 • Major public sector reforms introduced by Labour government from 1984 – saw some redrawing of the boundaries of the state and a reduction in analysis and evaluation of public expenditures • These reforms have been much studied internationally, but little copied • They reflect a narrow focus on outputs, as opposed to outcomes • Recent changes have strengthened the emphasis on outcomes and thus the role of analysis and evaluation

  4. Some General Issues in Monitoring and Evaluation • Who does it – internal or external ? • Role of specialist bodies - audit institutions, research institutes etc • Ex ante or ex post ? • What is the balance between monitoring as opposed to evaluation? • Covers all public expenditures or only “development” expenditures ? • Covers existing programs or activities or only new ones ? • How well is it integrated with the budgetary process ? • To what extent are economic concepts (use of scarce resources) built in • What are the incentives to undertake evaluation ? • What resources and skills are required for adequate evaluation capacity ?

  5. Stage I (A) - Early New Zealand PFM Reforms • Introduction of Planning Programming and Budgeting System (PPBS) from 1969 • Drawing on US approach – promotion of cost benefit analysis, cost-effectiveness analysis, program evaluation etc • Initially emphasis was on ex ante evaluation of capital projects e.g. • - new airports • - alternative land use (farming versus forestry) • - irrigation projects • And state investment in commercial undertakings – steel, energy etc • Need for ministry planning and evaluation units identified – to evaluate existing and new programs in terms of alternatives • Apart from capital projects neither monitoring nor evaluation took a firm hold – not closely aligned with budgetary cycle. • No real “challenge” function in Treasury (MOF) and therefore less incentive to undertake M and E

  6. Stage I (B) - Indicative Social and Economic Planning from 1969 • Developing emphasis on social policy research from early 1970s • National Development Conference 1969 – sectoral councils and coordination of government and private sector activities • National Development Council 1969 – 74 • Social Council and Social Development Council from 1973 – importance of evaluation of social programs - Social Impact Assessments (SIAs) of all major projects • Task Force on Economic and Social Planning from 1976 stressed the need for better and more evaluation of social programs • National Research Advisory Council – government coordination of all publicly funded “research” • Establishment of NZ Planning Council in 1974 – carried out medium term economic forecasting and suggested development and policy options. Stressed the need for more evaluation – of existing policies and programs, not just new ones • Evaluation activity coordinated through State Services Commission – its Review Division examined the efficiency and effectiveness of government ministries and agencies

  7. Stage II - Major Public Sector Reforms from 1984 • Major economic crisis at end of 1984 – foreign exchange crisis and unsustainable budget deficits and public debt • Objective - to reduce the role and size of the state – significant privatization and contracting out. Previous planning mechanisms abolished • Greater use of market mechanisms implied less need for research and evaluation – just follow the market • Treasury (MOF) suspicion of evaluation – no encouragement of it • A leaner, more efficient, public sector with civil servants placed on contracts • State Services Act 1988 and Public Finance Act 1989 provided the legal basis for the reforms • A reasonably rigorous intellectual model for the reforms – public choice and agency theory. Government has a purchaser and an ownership role in relation to its organizations • The state as a purchaser of services - ministers and chief executives negotiate an agreement for the purchase of outputs from ministries and agencies – quantity, quality, timing and cost

  8. Stage II - Major Public Sector Reforms from 1984 • These agreements form the basis for budget appropriations to produce output classes • i.e. budget funding provided on the basis of outputs to be produced • Ministers could purchase these outputs from outside the public service – i.e. contestability possibly leading to contracting out of service delivery • Outputs considered controllable by ministries and therefore an appropriate basis for such a “contractual” agreement • Therefore budget dialogue focused on needed outputs and operational efficiency issues • But recognition that outputs need to be properly specified – good practice guidance developed • And at what level should targets be set ? • Accrual accounting and budgeting introduced to reflect the full cost of inputs • Output pricing reviews facilitated cost control and consideration of market testing or contracting out

  9. Stage II - Major Public Sector Reforms from 1984 • Accountability and transparency through preparation by each ministry and agency of an annual statement of service performance – audited as part of the annual financial statements • Substantial devolution of financial and staffing decisions to chief executives – freedom to choose the type of inputs needed to produce the agreed outputs • Strong emphasis on financial controls to avoid over-spending • Capital expenditures handled by a separate process – part of the government’s ownership interest of safeguarding the value and capacity of its organizations • Separation of policy (outcomes) from service delivery (outputs), service delivery by autonomous agencies. Policy role in ministries. • Outcomes (effectiveness) not formally part of the budget dialogue • Outcomes perceived as the responsibility of ministers – they decide what outputs they require to achieve their desired outcomes • Separate personal performance agreement between the minister and head of each ministry and agency • Monitoring by management on the basis of outputs – which would reflect the work plans developed by each ministry or agency for its internal management

  10. Stage III (A) - Concerns emerged with the operation of this model • Ministers not really systematically interested in outcomes – or even outputs. Budget dialogue and purchase agreements became a “pro forma” exercise • Reflected in a weak link between outputs and funding – funding during the year provided on basis of agreed cash flows regardless of actual outputs produced and little feedback of previous years outputs into budget discussions of the following year • Loss of evaluation knowledge and skills in ministries and agencies – service delivery agencies should know a lot about the outcomes from the outputs they delivered yet this information was not being used • Lack of a focus on government-wide objectives – each minister operating separately. Each ministry or agency focuses only on its outputs. Concern about “atomization” of the public service

  11. Stage III(B) - Early Responses to the problem • From 1990 the government specified Strategic Results Areas (SRAs) and Key Results Areas (KRAs) to guide ministries and agencies and ministers in development of purchase agreement and to provide a “whole of government” approach • But too many KRAs and SRAs were developed. System later abolished. • Self evaluation by ministries and agencies included in chief executive’s performance agreement

  12. Stage III (C) - Current Developments (from 1999) • 2001 Review of the Centre concluded that evaluation was not conducted and used effectively • State Services Commission became leading proponent of enhancing evaluation e.g. project on improving the quality of policy advice 1999 • New managing for outcomes initiative introduced - whilst still budgeting for outputs. Provided for in Public Management Act of 2004. • Initiated by the civil service rather than by ministers – still low demand by many ministers for evaluation • Chief executives are not responsible for outcomes, but are responsible for managing to achieve outcomes • They also advise ministers on the possible outcomes to be pursued, their relative priority and which outputs may be most appropriate to achieve those outcomes

  13. Stage III (C) - Current Developments (from 1999) • Statements of Intent (SOI) – a public planning document prepared by each ministry and agency sets out goals over next 3-5 years. • It discusses outcomes and related outputs (the purchase interest) and maintaining capacity (the ownership interest). • Outcome and output information are expected to reflect information from evaluation, research and audit. • A continuous improvement cycle is operated. • It is presented to Parliament at the same time as the budget and must be endorsed by the responsible minister • Focus on enhancing the evaluation function in ministries and agencies • 2003 review suggested still problems concerning lack of an evaluation culture, inadequate evaluation capacity and poor cross-sector coordination

  14. Performance Auditing Role of the Office of the Auditor-General (OAG) • The independent external auditor of all government entities • Appointed by and reports to the Parliament. Public Expenditure Committee of Parliament follows up OAG reports. • Has a broad performance auditing mandate – to examine the extent to which a public entity is carrying out its duties effectively and efficiently • But a relatively small part of total OAG work – around 10 percent, although an increase is planned • OAG also audits performance information on outputs reported in annual statement of service performance produced by each entity – part of the annual financial statements. This work includes review of the performance targets • Has also been pro-active in encouraging improved performance reporting • Among the 12 performance audit reports in 2006 • - management of school property portfolio • - effectiveness of certain housing programs • - police anti-burglary work • - administration of grants by certain ministries

  15. Role of the Education Review Office • A specialist evaluation office established in 1990 • ERO evaluates and reports on the quality of education delivered by individual schools and early childhood education services • By 2006/07 ERO will be reviewing schools on average every three years • ERO reviews in schools have the following three strands of investigation: • - School Specific Priorities, mainly level of student achievement • and learning • - Areas of National Interest – special topics chosen by ERO: • emotional safety; student attendance; stand downs, suspensions • and exclusions; and teacher registration.

  16. Where New Zealand Is Now • The formal structures of the original reforms remain – viz budgeting on the basis of outputs, chief executive performance agreements, accrual accounting and budgeting, managerial devolution, reporting through statements of service performance - remain in place • Development of an outcomes focus has been slow. Some lack of support from ministers. Difficulties in linking outputs with outcomes. • However resources devoted to evaluation appear to be increasing. Australian Evaluation Society has played an important role as have non-governmental research groups and the universities. • Major resource reallocations or policy developments are still done outside the formal budgetary process – reflecting formal focus of budget still being on outputs

  17. Where New Zealand Is Now • Budget discussions do not generally involve review of existing programs. Often new spending proposals are a short-term response to political issues • Overall there remains a weak link between monitoring and evaluation and the budget process • Chief executive performance agreements have been more important in promoting evaluation • The outputs process still does not in general reflect the way ministries and agencies manage themselves internally • However there is considerable variation in progress made by different ministries and agencies • But the SOI direction is promising

More Related