The State of Program Evaluation in the Canadian Federal Government

1. The State of Program Evaluation in the Canadian Federal Government Glenn Wheeler Director, Results Measurement Office of the Auditor General of Canada October 17, 2001

2. Outline of Presentation 1. Background on Canadian System of Government 2. About the Office of the Auditor General of Canada 3. The Auditor General’s Approach to Evaluation 4. The Auditor General’s Audits of Program Evaluation 5. Emergence of Performance Reporting within the Context of Program Evaluation 6. Lessons Learned from the Canadian Experience

3. 1. Background on Canadian System of Government Division of Powers - a sovereign democracy: - federation of 10 provinces and 3 territories - a federal Parliament and 13 provincial/territorial legislatures - federal government - authority in all matters not specifically reserved for provincial/territorial governments - provincial/territorial governments - authority for education, hospitals, property and local government

4. Federal Government - over 100 departments and agencies responsible for areas such as employment, national defence, fisheries, agriculture and finance - each department and agency has a minister who reports to Parliament - budgets approved annually

5. 2. About the Office of the Auditor General of Canada History 1878: Office of the Auditor General established 1931: Clarification of Auditor General’s role 1950: beginning of expansion of Auditor General’s mandate 1977: new legislation - the Auditor General Act 1994: legislation amended to allow Auditor General to allow for tabling of up to 3 reports per year in addition to annual report 1995: legislation amended to establish a Commissioner of the Environment and Sustainable Development within the Office

6. Current Office - new Auditor General, Sheila Fraser, for 10 year term - headquarters in Ottawa with 5 regional offices - staff of 520 auditors and support staff Role - The Auditor General aids accountability by conducting independent audits of federal government operations - these audits provide members of Parliament with objective information to help them examine the government’s activities and hold it to account

7. The Auditor General’s Role

8. Vision - We are committed to making a difference for Canadians by promoting, in all our work for Parliament, answerable, honest and productive government that reflects a commitment to sustainable development Mission - The Office conducts independent audits and examinations that provide information, advice and assurance to Parliament. We promote accountability and best practices in government operations.

9. Auditor General Act Section 7 (2) of the Auditor General Act requires the Auditor General to call attention to anything he considers to be of significance and of a nature that should be brought to the attention of Parliament, including cases in which he has observed: a) accounts have not been properly maintained or public money has not been fully accounted for; b) essential records have not been maintained to ensure proper allocation of revenue and to ensure that expenditures have been made only as authorized; c) money has been spent other than for the purposes for which it was appropriated by Parliament;

10. d) money has been expended with due regard to economy and efficiency; e) satisfactory procedures have not been established to measure and report on the effectiveness of programs; or f) money has been expended without due regard to effects on the environment.

11. 3. The Auditor General’s Approach to Program Evaluation • Types of audits • - attest audits • - compliance audits • - value-for-money audits • A value-for-money audit is a systematic and objective examination of government activities. Its scope includes the examination of economy, effficiency and cost-effectiveness of activities. It also examines procedures to measure effectiveness. The subject of the audit can be a program, sectoral activity, or a government-wide sectional area. • A program evaluation is a disciplined assessment of government programs and activities. It is based on independent, systematic measurement and analysis, carried out to meet expectations set in policy and standards, and publicly reported.

12. Interpretation - although value-for-money audits and program evaluations are similar, they are different in some ways - the Auditor General does not conduct program evaluations but rather value-for-money audits which examine procedures used by departments to measure effectiveness (i.e., to undertake program evaluations)

13. What the Auditor General expects when undertaking audits on program evaluation Value-for-money audits examine how departments plan and undertake program evaluations of their programs, specifically: 1) the department’s evaluation policy 2) responsibility for program evaluation within the department 3) the organization of the evaluation branch within the department 4) resources (people and funds) dedicated to program evaluation 5) the number and quality of program evaluations completed 6) the mechanisms used to ensure quality of program evaluations 7) the relevance and timeliness of program evaluations 8) how program evaluation findings are used

14. Why the Auditor General considers program evaluation to be a crucial function in the federal government 1) to provide information to allow senior management to better manage their programs 2) to provide evidence for departments concerning whether to continue funding existing programs or re-allocate these funds to other programs 3) to provide information to managers as these seek funding to renew programs 4) to encourage accountability to Parliament for the results of programs 5) to enhance public debate surrounding policy and program options 6) to allow government to determine priorities 7) to encourage transparency in departments

15. 4. The Auditor General’s Audits of Program Evaluation Audits undertaken on the program evaluation function across the federal government - 1978 - first audit of evaluation function 1983 - update on implementation of 1978 recommendations 1993 - major government-wide audit on the state of program evaluation in the federal government 1996 - examination of program evaluation in the context of a changing public sector environment 2000 - reporting performance to Parliament

16. Federal Government Policy on Program Evaluation 1977 - original policy 1991 - revised policy 1994 - review policy (review, internal audit and program evaluation 1995 - revised expenditure management system 2001 - new evaluation policy

17. 1993 Audit - Key Observations - program evaluation branches poorly funded, and often subsumed within internal audit banches - departments did not plan for evaluation work - many program evaluations of smaller programs; fewer evaluations of larger more important programs - program evaluations often focussed on operational issues and less likely to challenge the existence of a program - limited use of program evaluation results - program evaluation results not linked to resource allocation or accountability reporting

18. 1996 Audit - Key Observations - program evaluation did not consistently provide effectiveness information for government decision making and broader accountability - more larger programs were being evaluated than in 1993, but undertaken in response to external demands - scope and quality of program evaluation remained a concern

19. 5. Emergence of Performance Reporting in Context of Program Evaluation Two key developments 1) revised expenditure management system - in 1995 the federal government announced that under its revised expenditure management system departments would be required to submit an annual “Performance Report” to Parliament 2) creation of new agencies - since 1997 three new agencies have been created to streamline key activities previously undertaken by many departments (Canadian Food Inspection Agency, Parks Canada, Canada Customs and Revenue Agency) - new agencies required to submit annual reports - legislation for the three agencies states that the Auditor General annually shall “provide an assessment of the fairness and reliability of the information about the Agency’s performance”

20. 2000 Audit - Key Observations - department focus too much on listing program activities and outputs and to little on reporting outcomes or results - reporting makes too little use of evaluation findings on the results that activities are accomplishing and on how programs are contributing to outcomes

21. 6. Lessons Learned from the Canadian Experience Organizations must: 1) have a unit in place with resources (people and funds) and expertise to undertake program evaluations 2) have a reporting structure in place so that the manager of the program evaluation unit reports directly to the head of the organization or its senior management committee 3) consider constructing a policy formalizing roles and responsibilities with respect to program evaluation 4) plan to undertake program evaluations. Plans should cover a one-year period at a minimum, and several years preferably 5) have a plan which prioritizes which programs should be evaluated based on their importance and level of risk

22. 6) have mechanisms built into the process to ensure the quality of program evaluations and to ensure that they are timely 7) ensure that program evaluation results are utilized by senior management in making decisions regarding funding and whether programs should be redesigned or discontinued 8) follow up with the management of programs evaluated to determine whether evaluation recommendations are being implemented 9) periodically review the performance of their program evaluation units to determine the value that program evaluation adds to the organization