Developing Evaluation Capacity

1. Developing Evaluation Capacity EEN [Canada] Forum Shelley Borys Director, Evaluation September 30, 2010

2. Context • The importance of building evaluation capacity is linked to the following factors: • Increased requirements for evaluation • The requirements under the new Policy on Evaluation to move towards coverage of 100% of both transfer program and direct program spending on a five-year cycle. • The increased interest in the use of evaluation evidence to provide useful feedback regarding program improvement, to demonstrate progress toward outcomes, and to support decision-making.

3. Overview • This presentation will discuss ways in which Environment Canada is working to increase its own evaluation capacity. • This topic will be explored from three perspectives: • How we build evaluation capacity within the Evaluation Division; • Activities to increase the capacity for programs to be evaluated; and • How we build capacity to use evaluation.

4. Evaluation Competencies • Two frameworks exist which elaborate a series of evaluator competencies which can serve as a guide to capacity development: • Canadian Evaluation Society Credentialing Initiative • Five competency domains: reflective (7), technical (16), situational (9), management (7), interpersonal (10) • Treasury Board Secretariat Centre for Excellence in Evaluation • Four competency clusters: intellectual (3), management (2), relationship (3), personal (2)

5. Building Evaluator Capacity:Environment Canada Evaluation Model • Intent is to conduct as much evaluation work as is possible in-house • Use contractors for evaluations that are technically beyond our capacity, interdepartmental evaluations, and where we have contract dollars but not staff time • This helps build evaluation capacity by: • Having evaluators gain hands-on evaluation experience • Helping evaluators become subject-matter experts • Facilitating evaluators to develop a network of contacts within the Department

6. Building Evaluator Capacity:Project Teams • Evaluation Division teams are formed for each project, with: • A project lead, who coordinates communications, directs the work of team members, and is the lead pen on deliverables; • Team members, who support the efforts of the project lead in performing evaluation activities and preparing reports. • A peer reviewer, who is another evaluation manager whose role is to review in-progress deliverables, before they are submitted for management review; and • Teams are not necessarily hierarchical. • This helps build evaluation capacity by • Providing exposure to different styles of evaluation and project management • Providing an opportunity to work on different tasks and a range of projects • Enabling evaluators to take on additional responsibilities while managing risk.

7. Building Evaluator Capacity:Templates • Evaluation projects use a common file folder structure and basic document formats. • These templates build evaluation capacity by: • Ensuring that all required issues are addressed; • Laying out a somewhat standard process for evaluation; • Reducing the risk of uneven evaluation conduct.

8. Building Evaluator Capacity:Generic Evaluation Framework • One templated item of particular importance is the generic evaluation framework. • This is a matrix table contained within the evaluation plan template that sets out: • Key questions to be asked; • Indicators for each question; • Sources and methods that would support the analysis; and • The specific Treasury Board Secretariat Policy on Evaluation issue each question relates to. • This evaluation framework helps build evaluator capacity by: • Ensuring that all evaluations begin from the same foundation • Serving as guidance around the identification of indicators and data sources.

9. Building Evaluator Capacity:Training • Staff within the Evaluation Division are encouraged to seek out opportunities for training and professional development • Conferences, networking events, lunch and learn sessions • Resource library • This helps to build evaluation capacity by: • Helping evaluation staff remain abreast of developments within the theory and practice of evaluation

10. Building Capacity to be Evaluated: Shadow Plan • Two years in advance of a planned evaluation, we conduct a mini evaluability assessment: • Performance measurement? • Logic Model? • Key issues of interest? • Timing consideration? • Intent is that improvements can be introduced before the evaluation starts, thereby strengthening the evaluation • Performance measurement • Relevance to managers

11. Building Capacity to be Evaluated:Performance Measurement Improvements • The Evaluation Division regularly assesses the quality of performance measurement in its evaluations, as shown in the following table: • In recommending improvements to performance measurement, subsequent evaluations will have better data to draw upon.

12. Building Capacity to be Evaluated and to Use Evaluation: Integration with Program Design • The Department has established several mechanisms for the Division to provide feedback on potential programs at an early stage. The Division is consulted on: • Memoranda to Cabinet submissions; • Treasury Board Secretariat submissions; and • The design of performance measurement strategies. • This helps build the capacity to be evaluated and to use evaluation by: • Establishing a dialogue between program staff and evaluators in advance of evaluation; • Ensuring evaluation results are considered in the design of new initiatives; and • Ensuring that program design supports evaluation e.g. defined outcomes, performance measurement.

13. Building Capacity to Use Evaluation:Management Response Guidelines • The Evaluation Division has prepared guidelines for management to draw upon when preparing their response to evaluation recommendations. • These guidelines describe the essential components of a management response: • In particular, the development of specific actions with deadlines and clear responsibilities. • This helps build the capacity to use evaluation by: • Ensuring the clarity of management responses; and • Facilitating the measurement of progress against action plans.

14. Building Capacity to Use Evaluation:Follow-up Process • After evaluation reports and the management responses to their recommendations have been approved, the Department works to ensure that these recommendations are followed through on. • This is done formally at the Departmental Evaluation Committee meetings, where the status of actions stemming from past evaluations is a standard agenda item. • This helps build the capacity to use evaluation by: • Formalizing the implementation process; • Ensuring that senior management remain aware of outstanding action items; and • Providing a forum for any barriers to implementation to be discussed and responded to.

15. Building Capacity to Use Evaluation:Management Support • A successful evaluation unit requires a suitably enabling environment. • At Environment Canada, senior management understanding and support of evaluation is evident. • Departmental Evaluation Committee is active and engaged • This helps build evaluation capacity by: • Signalling to program personnel that evaluation is seen as a furtherance of departmental work – not an interruption; and • Strengthening the connection between evaluation evidence and senior decision-making.

16. Evaluator Capacity In-house Conduct Team Structure Templates Generic Evaluation Framework Training Capacity to be Evaluated / Use Evaluation Performance Measurement Improvements Shadow Plan Integration with Program Design Management Response Guidelines Follow-up Process Management Support Summary of Practices