170 likes | 274 Views
Monitoring and Evaluation in New Zealand Paper for NDRC Monitoring and Evaluation Conference Beijing, 25-26 October, 2006. David Shand dshand@worldbank.org. The New Zealand Context. A small country – 4.2 million people, 15 percent indigenous population
E N D
Monitoring and Evaluation in New ZealandPaper for NDRC Monitoring and Evaluation Conference Beijing, 25-26 October, 2006 David Shand dshand@worldbank.org
The New Zealand Context • A small country – 4.2 million people, 15 percent indigenous population • Middle level income – 28th in world, GDP per head of US$25,000 • “Westminster” form of government, reflecting British colonial heritage. Strong executive government but role of Parliament has increased following change in electoral system • Tradition of a strong central government and a progressive social welfare tradition • Key objectives set by the current government • - to strengthen national identity and uphold the principles of • the Treaty of Waitangi; • - to grow an inclusive, innovative economy for the benefit of all; • - to maintain trust in government and provide strong social • services; • - to improve New Zealanders’ skills; • - to reduce inequalities in health, education, employment and • housing; • - to protect and enhance the environment
The Development of Monitoring and Evaluation • Two major strands or initiating factors • - Public Financial Management (PFM) reforms • - Social science research emphasis • Program analysis and evaluation with the introduction of Planning Programming Budgeting system (PPBS) from late 1970 • Indicative National Planning system from 1969 to about 1980 • Major public sector reforms introduced by Labour government from 1984 – saw some redrawing of the boundaries of the state and a reduction in analysis and evaluation of public expenditures • These reforms have been much studied internationally, but little copied • They reflect a narrow focus on outputs, as opposed to outcomes • Recent changes have strengthened the emphasis on outcomes and thus the role of analysis and evaluation
Some General Issues in Monitoring and Evaluation • Who does it – internal or external ? • Role of specialist bodies - audit institutions, research institutes etc • Ex ante or ex post ? • What is the balance between monitoring as opposed to evaluation? • Covers all public expenditures or only “development” expenditures ? • Covers existing programs or activities or only new ones ? • How well is it integrated with the budgetary process ? • To what extent are economic concepts (use of scarce resources) built in • What are the incentives to undertake evaluation ? • What resources and skills are required for adequate evaluation capacity ?
Stage I (A) - Early New Zealand PFM Reforms • Introduction of Planning Programming and Budgeting System (PPBS) from 1969 • Drawing on US approach – promotion of cost benefit analysis, cost-effectiveness analysis, program evaluation etc • Initially emphasis was on ex ante evaluation of capital projects e.g. • - new airports • - alternative land use (farming versus forestry) • - irrigation projects • And state investment in commercial undertakings – steel, energy etc • Need for ministry planning and evaluation units identified – to evaluate existing and new programs in terms of alternatives • Apart from capital projects neither monitoring nor evaluation took a firm hold – not closely aligned with budgetary cycle. • No real “challenge” function in Treasury (MOF) and therefore less incentive to undertake M and E
Stage I (B) - Indicative Social and Economic Planning from 1969 • Developing emphasis on social policy research from early 1970s • National Development Conference 1969 – sectoral councils and coordination of government and private sector activities • National Development Council 1969 – 74 • Social Council and Social Development Council from 1973 – importance of evaluation of social programs - Social Impact Assessments (SIAs) of all major projects • Task Force on Economic and Social Planning from 1976 stressed the need for better and more evaluation of social programs • National Research Advisory Council – government coordination of all publicly funded “research” • Establishment of NZ Planning Council in 1974 – carried out medium term economic forecasting and suggested development and policy options. Stressed the need for more evaluation – of existing policies and programs, not just new ones • Evaluation activity coordinated through State Services Commission – its Review Division examined the efficiency and effectiveness of government ministries and agencies
Stage II - Major Public Sector Reforms from 1984 • Major economic crisis at end of 1984 – foreign exchange crisis and unsustainable budget deficits and public debt • Objective - to reduce the role and size of the state – significant privatization and contracting out. Previous planning mechanisms abolished • Greater use of market mechanisms implied less need for research and evaluation – just follow the market • Treasury (MOF) suspicion of evaluation – no encouragement of it • A leaner, more efficient, public sector with civil servants placed on contracts • State Services Act 1988 and Public Finance Act 1989 provided the legal basis for the reforms • A reasonably rigorous intellectual model for the reforms – public choice and agency theory. Government has a purchaser and an ownership role in relation to its organizations • The state as a purchaser of services - ministers and chief executives negotiate an agreement for the purchase of outputs from ministries and agencies – quantity, quality, timing and cost
Stage II - Major Public Sector Reforms from 1984 • These agreements form the basis for budget appropriations to produce output classes • i.e. budget funding provided on the basis of outputs to be produced • Ministers could purchase these outputs from outside the public service – i.e. contestability possibly leading to contracting out of service delivery • Outputs considered controllable by ministries and therefore an appropriate basis for such a “contractual” agreement • Therefore budget dialogue focused on needed outputs and operational efficiency issues • But recognition that outputs need to be properly specified – good practice guidance developed • And at what level should targets be set ? • Accrual accounting and budgeting introduced to reflect the full cost of inputs • Output pricing reviews facilitated cost control and consideration of market testing or contracting out
Stage II - Major Public Sector Reforms from 1984 • Accountability and transparency through preparation by each ministry and agency of an annual statement of service performance – audited as part of the annual financial statements • Substantial devolution of financial and staffing decisions to chief executives – freedom to choose the type of inputs needed to produce the agreed outputs • Strong emphasis on financial controls to avoid over-spending • Capital expenditures handled by a separate process – part of the government’s ownership interest of safeguarding the value and capacity of its organizations • Separation of policy (outcomes) from service delivery (outputs), service delivery by autonomous agencies. Policy role in ministries. • Outcomes (effectiveness) not formally part of the budget dialogue • Outcomes perceived as the responsibility of ministers – they decide what outputs they require to achieve their desired outcomes • Separate personal performance agreement between the minister and head of each ministry and agency • Monitoring by management on the basis of outputs – which would reflect the work plans developed by each ministry or agency for its internal management
Stage III (A) - Concerns emerged with the operation of this model • Ministers not really systematically interested in outcomes – or even outputs. Budget dialogue and purchase agreements became a “pro forma” exercise • Reflected in a weak link between outputs and funding – funding during the year provided on basis of agreed cash flows regardless of actual outputs produced and little feedback of previous years outputs into budget discussions of the following year • Loss of evaluation knowledge and skills in ministries and agencies – service delivery agencies should know a lot about the outcomes from the outputs they delivered yet this information was not being used • Lack of a focus on government-wide objectives – each minister operating separately. Each ministry or agency focuses only on its outputs. Concern about “atomization” of the public service
Stage III(B) - Early Responses to the problem • From 1990 the government specified Strategic Results Areas (SRAs) and Key Results Areas (KRAs) to guide ministries and agencies and ministers in development of purchase agreement and to provide a “whole of government” approach • But too many KRAs and SRAs were developed. System later abolished. • Self evaluation by ministries and agencies included in chief executive’s performance agreement
Stage III (C) - Current Developments (from 1999) • 2001 Review of the Centre concluded that evaluation was not conducted and used effectively • State Services Commission became leading proponent of enhancing evaluation e.g. project on improving the quality of policy advice 1999 • New managing for outcomes initiative introduced - whilst still budgeting for outputs. Provided for in Public Management Act of 2004. • Initiated by the civil service rather than by ministers – still low demand by many ministers for evaluation • Chief executives are not responsible for outcomes, but are responsible for managing to achieve outcomes • They also advise ministers on the possible outcomes to be pursued, their relative priority and which outputs may be most appropriate to achieve those outcomes
Stage III (C) - Current Developments (from 1999) • Statements of Intent (SOI) – a public planning document prepared by each ministry and agency sets out goals over next 3-5 years. • It discusses outcomes and related outputs (the purchase interest) and maintaining capacity (the ownership interest). • Outcome and output information are expected to reflect information from evaluation, research and audit. • A continuous improvement cycle is operated. • It is presented to Parliament at the same time as the budget and must be endorsed by the responsible minister • Focus on enhancing the evaluation function in ministries and agencies • 2003 review suggested still problems concerning lack of an evaluation culture, inadequate evaluation capacity and poor cross-sector coordination
Performance Auditing Role of the Office of the Auditor-General (OAG) • The independent external auditor of all government entities • Appointed by and reports to the Parliament. Public Expenditure Committee of Parliament follows up OAG reports. • Has a broad performance auditing mandate – to examine the extent to which a public entity is carrying out its duties effectively and efficiently • But a relatively small part of total OAG work – around 10 percent, although an increase is planned • OAG also audits performance information on outputs reported in annual statement of service performance produced by each entity – part of the annual financial statements. This work includes review of the performance targets • Has also been pro-active in encouraging improved performance reporting • Among the 12 performance audit reports in 2006 • - management of school property portfolio • - effectiveness of certain housing programs • - police anti-burglary work • - administration of grants by certain ministries
Role of the Education Review Office • A specialist evaluation office established in 1990 • ERO evaluates and reports on the quality of education delivered by individual schools and early childhood education services • By 2006/07 ERO will be reviewing schools on average every three years • ERO reviews in schools have the following three strands of investigation: • - School Specific Priorities, mainly level of student achievement • and learning • - Areas of National Interest – special topics chosen by ERO: • emotional safety; student attendance; stand downs, suspensions • and exclusions; and teacher registration.
Where New Zealand Is Now • The formal structures of the original reforms remain – viz budgeting on the basis of outputs, chief executive performance agreements, accrual accounting and budgeting, managerial devolution, reporting through statements of service performance - remain in place • Development of an outcomes focus has been slow. Some lack of support from ministers. Difficulties in linking outputs with outcomes. • However resources devoted to evaluation appear to be increasing. Australian Evaluation Society has played an important role as have non-governmental research groups and the universities. • Major resource reallocations or policy developments are still done outside the formal budgetary process – reflecting formal focus of budget still being on outputs
Where New Zealand Is Now • Budget discussions do not generally involve review of existing programs. Often new spending proposals are a short-term response to political issues • Overall there remains a weak link between monitoring and evaluation and the budget process • Chief executive performance agreements have been more important in promoting evaluation • The outputs process still does not in general reflect the way ministries and agencies manage themselves internally • However there is considerable variation in progress made by different ministries and agencies • But the SOI direction is promising