1 / 13

How Evaluations are Done at NRCan

Learn about the planning, costing, and conducting of evaluations at NRCan's Strategic Evaluation Division, including the evaluation cycle, assessment methods, and contracting process.

allens
Download Presentation

How Evaluations are Done at NRCan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Evaluations are Done at NRCan Strategic Evaluation Division Natural Resources Canada

  2. Purpose • To provide information on how evaluations are planned, costed and conducted once they are on the five-year departmental Evaluation Plan.

  3. The Cycle for Evaluation Reports START Evaluation Assessment (2-3 months) Implementing Change/Follow Up Terms of Reference DEC Approval Internet Posting (2-3 months) Large evaluations typically take 12-18 months to complete. Some phases may overlap. Contracting (3-4 months) Management Responses (1 month) Field Work/ Analysis (6-8 months) Report & Recommendations (2-4 months)

  4. Evaluation Assessment • Evaluation Assessments are prepared based on: • program profiles (e.g., objectives; logic models; organization and governance; expenditures; an assessment performance information); • the calibration of the evaluation based on risk criteria, as part of overall 5-year Plan • (i.e., program renewal; materiality; context and need; visibility; management practices and structure; policy and delivery complexity; performance measurement; and past evaluation and audit findings); • the evaluation questions (TBS Policy identifies the generic questions and these may be supplemented by others identified by Program); • an initial plan on how to conduct the evaluation (i.e., methods to be used and levels of effort; contracting strategy; timelines; and estimated costs). • Evaluation Assessments are developed based on experience and professional judgement.

  5. Scope/Costing Considerations • Based on evaluation assessment, the scope and costing are developed, taking into account: • TBS policy’s minimum standards and required "multiple lines of evidence" (e.g., documents; interviews; administrative data; survey results; case studies; focus groups). • SED’s internal risk assessment based on additional methods support quality, rigour & richness of the evaluation. • Sufficient level of evidence to be collected to allow conclusions to be drawn on each program (especially G&C programs). • There are many risk-factors (previous page) and considerations, e.g.: • number of programs, availability of data, past evaluations; • enough interviewees to ensure the full story; • in-person interviews are always better than over the phone • surveys provide less in-depth information from many people; • case studies provide in-depth knowledge of one project and how projects contribute to the achievement of program objectives. • Need for contracting

  6. Contracting • Contractors may be used as part of the evaluation team, based on: • The need for subject matter expertise; • Internal capacity and timing considerations; and • The need for third party (non-NRCan) involvement • Value for money (optimal mix of internal and external resources) is determined through: • The availability of internal staff to meet coverage requirements; and • The competitive procurement process designed to achieve the best value for the Crown • Evaluation has aSupply Arrangement with nine evaluation firms* based on a competitive process to qualify the firm and contractors. • Based on the TOR, an RFP is sent to a minimum of 4 firms. • Contracting takes at least 3 months from SOW to signed contract. • Responses to the RFPs, provide suggestions for altering the SOW and what the company is prepared to do for the available budget. • If a subject matter expert is required, Evaluation will contract directly with the expert to be part of the team (e.g., nuclear expertise). * KPMG; Science Metrix; PMN; PRA; Goss Gilroy;TDV Global; Boulton; Baastel; and CPM.

  7. Terms of Reference • The TOR are derived from the Evaluation Assessment and scoping/costing analysis. • TOR must be approved by the Evaluation Committee. • TOR include: • overview of the entity being evaluated; • evaluation issues and questions, • methods (e.g., interviews; surveys) to be used; • contracting approach (i.e., in house; contracted out; or hybrid); • timelines; • resources required (e.g., estimated contract cost); and • governance (e.g., a working group of program and evaluation officials).

  8. Field Work • Evaluators work on several projects simultaneously. • Field work presents many challenges: • field work requires the input of programs; • contact information for surveys & interviewees not always readily available; • unplanned delays are very difficult for contractors to manage; • work around seasonal cycles: e.g. interviewees not available during the summer; contractors and programs are extremely busy prior to March 31. • Technical reports are usually produced for each method (e.g., interviews). All information is analyzed by evaluation question. • Preliminary findings are presented to programs to confirm findings and seek any additional information.

  9. Report and Recommendations • Preliminary findings serve to highlight and validate key issues from collected evidence, and provide basis for the outline for the report. • Based on the findings, the report is written and the recommendations are developed. • The draft report is vetted with the Program and discussions take place on the recommendations (do they flow from findings; do they make sense; can they be implemented). • Report length is influenced by complexity of subject (e.g., one program or many) and need to present evidence. • Management responses and action plan are drafted by the program and ADM approval is sought.

  10. Approval & Posting of Reports • The DEC advises the Deputy Minister (DM) on the report and management responses action plans. The DM must approve. • DM-approved reports are provided to TBS and may be examined for quality during the annual MAF assessment of evaluation. • The Evaluation Policy requires, complete, approved evaluation reports along with management responses and action plans to be posted in both official languages in a timely manner (i.e., 90 days). • Reports are reviewed by ATIP and Communications. • If required, Communications and the program prepare media lines.

  11. Management Responses & Follow Up • ADMs are responsible for implementing the action plans for each recommendation. • The Evaluation Division follows up with Sectors on the implementation of action plans and updates DEC. • DEC decides if the management responses have been satisfactorily implemented and when the file can be closed. • The activity/program is re-considered as part of Evaluation Plan in the following evaluation cycle.

  12. What Does Evaluation Add? • Accountability (posted reports and input into DPRs) • Strategic input into program decision-making and development (e.g., assistance in developing logic models and performance measurement strategies) • including information for MCs and TB Submissions • Neutral perspectives and additional data collection methods • Evidence for internal reviews (Strategic Reviews/SOR etc.) • Corrective change, starting before the evaluation is over • Follow up on recommendations

  13. Conclusions • We hope this presentation has provided a better understanding of how evaluations are planned, costed and conducted. • Questions? send to dvinette@nrcan.gc.ca

More Related