Overview of Evaluation at NRCan

1. Evaluation at NRCan:Information for Program ManagersStrategic Evaluation DivisionScience & Policy IntegrationJuly 2012

2. Purpose • The purpose of this document is to provide program managers with an overview of the evaluation function at NRCan. • The TBS Policy on Evaluation (April 2009) requires that all direct spending, including all G&C Programs, be evaluated every five years • most program managers will find themselves participating in an evaluation at some point.

3. What is Evaluation? • Evaluations are the systematic collection and analysis of evidence on the outcomes of programs to make judgments about • their relevance; • performance; and • alternative ways to deliver them or to achieve the same results.” • Evaluations must be neutral, and evidence-based. • An evaluation is not the same as an audit.

4. Why do Evaluations? • The objective of evaluation is to create a comprehensive and reliable base of evidence to support: • policy and program improvement; • expenditure management; • Cabinet decision-making; and • public accountability. • Evaluations are often required to support TB submissions and Memoranda to Cabinet. • They are also a critical source of evidence for Strategic Reviews, to support resource reallocation (next review will occur in 2014).

5. Background: Evaluation Stakeholders Evaluation Reports Policy & Program Improvement Expenditure Management/ Cabinet Decision-making Accountability/ Public Reporting Cabinet Parliament NRCan DM Strategic Review Canadian Public Sector ADMs Program Managers TBS The challenge is to meet the diverse information needs of many stakeholders for decision-making AND public accountability in a timely fashion.

6. The Evaluation Cycle Evaluation Planning Evaluation Assessment (1-3 months) Implementing Change Including planning and approvals/posting, large evaluations typically take 12-18 months to complete. Some phases may overlap. Approvals/ Posting (2-3 months) Contracting (1-2 months) Field Work/Analysis (6-8 months) Management Responses (1 month) Report & Recommendations (1-4 months)

7. Evaluation Planning • NRCan must evaluate all direct program spending, including all ongoing grant and contribution programs, every five years. • NRCan has developed a five-year Evaluation Plan based on PAA units that is updated annually and approved by the Evaluation Committee. • In most cases, the evaluation of an individual program will be conducted within the scope of a broader evaluation of a PAA unit. • The current plan summary appears on the Strategic Evaluation Internet site at: http://nrcan.gc.ca/evaluation/plans-eng.php.

8. Phases of an Evaluation • Evaluation Assessment (1-3 months) • research and planning to understand the programs • develop the Terms of Reference • obtain approval from the Evaluation Committee • Contracting (1-2 months) • Consultants are often used to supplement in-house staff. Their roles will vary by project. • Fieldwork or Data Collection/Analysis (6-8 months) • develop a detailed methodology report • methodologies: key informant interviews; focus groups; file/document/literature reviews; surveys; case studies; and data and economic analysis • analyse information collected from these multiple lines of evidence to develop conclusions

9. Phases of An Evaluation, cont’d • Reporting & Development of Recommendations (2-4 months) • prepare preliminary findings and discuss with programs • draft report • address comments and revisions • develop recommendations • Management Responses (1 month) • obtain ADM-approved management responses and action plans to the recommendations • Approvals/Posting of report (2-3 months) • recommendation by the Evaluation Committee • approval by the DM • translation, ATIP review, media lines, release on Internet

10. Evaluation Questions and Issues • Evaluations address relevance and performance. • Relevance issues focus on: • continued need for program; • alignment with government priorities; and • alignment with federal roles and responsibilities. • Performance issues focus on effectiveness: • achievement of expected outcomes; and • demonstration of efficiency and economy. • Evaluators work with program managers to develop more detailed evaluation questions relevant to their program.

11. Roles and Responsibilities • Under the TBS Evaluation Policy, Deputy Ministers are responsible for the evaluation function. • NRCan’s Departmental Evaluation Committee – an ADM-level Committee – is chaired by the DM • NRCan’s Head of Evaluation – who is also the DG of Planning and Performance Management Reporting – reports to the Evaluation Committee.

12. The Role of the Evaluation Division • The Strategic Evaluation Division (SED) is responsible for: • Proposing a five year departmental evaluation plan to the Evaluation Committee, and updating it annually; • Managing and conducting evaluation studies, including managing contracts and deliverables when consultants are used and issuing reports in a timely manner. • Additionally, SED will help program managers develop their performance measurement strategies, with the goal of ensuring that good data is collected to support future evaluations. • Evaluation will work with your team to develop objectives, a logic model, a performance measurement framework and evaluation requirements. • NRCan's Strategic Evaluation Division is also responsible for reviewing and providing advice on the accountability and performance provisions in Cabinet documents (Memoranda to Cabinet (MCs) and TB Submissions).

13. The Role of Program Managers • Program managers are key to conducting evaluations. • They are responsible for developing, implementing and monitoring ongoing performance measurement – the foundation of evaluation. • Additionally, during an evaluation, they must be actively involved in: • explaining how their programs work; • contributing to evaluation planning, including identifying more detailed evaluation questions; • providing performance measurement information on resources used, activities undertaken and results achieved; • providing detailed documentation (see next slide) and suggestions on potential interviewees, case studies etc.; • participating in working groups to review questionnaires, preliminary findings, draft evaluation reports, etc.; • developing management responses and action plans for their ADMs and implementing them after the evaluation.

14. Key Documents for An Evaluation • In preparation for an evaluation, program managers will be asked to provide key documents as early as possible, including: • Legislation, Regulations, MCs, TB Submissions • RMAFs, RBAFs or Performance Measurement Frameworks • references in budgets, SFTs, DPRs, RPPs • briefing notes, reports --including annual and project reports, studies, databases • websites and communications products • five-years of financial expenditures for the PAA unit including G&C expenditures, O&M, and salaries

15. Questions and Assistance • If you have any questions on evaluation, or wish evaluation assistance in developing performance measurement information or Cabinet documents, please contact: • The Director of Strategic Evaluation (613) 996-9649 • Electronic copies of this document, completed evaluation reports and the Terms of Reference for the Evaluation Committee are available at: http://nrcan.gc.ca/evaluation/index-eng.php