Prepared by Government Consulting Services for Audit and Evaluation Branch, Environment Canada

1. Clean Air Agenda Horizontal Evaluation Roll-up: Synopsis of Evaluation Results Available Up to September 2010 Final Prepared by Government Consulting Services for Audit and Evaluation Branch, Environment Canada November 8, 2010

2. Acronyms

3. Introduction • This report has been prepared in fulfillment of a commitment to produce a horizontal roll-up of the results of the evaluations done for the Clean Air Agenda (CAA) themes. • The roll-up is intended to help participating departments and agencies report on their CAA results to Parliamentarians and Canadians, and to inform decisions regarding CAA funding and activities beyond 2011. • The report comprises four sections: • CAA Profile; • Methodology; • Findings; and • Observations. Acknowledgements: • The project team was led by Martine Perrault under the direction of the Environment Canada Evaluation Director, Shelley Borys, and included Linda Lee, Gavin Lemieux and Katheryne O’Connor. The team would also like to thank all the individuals from the following organizations who participated in the working group for this project and well as contributed to the development of the evaluation reports that form the basis for this horizontal roll-up: Environment Canada (EC) (including the Clean Air Agenda - Results Management Secretariat), Department of Foreign Affairs and International Trade (DFAIT), Health Canada (HC), Indian and Northern Affairs Canada (INAC), National Research Council (NRC), National Resources Canada (NRCan), Public Health Agency of Canada (PHAC), and Transport Canada (TC). This report was prepared for Environment Canada by James Coflin from Government Consulting Services (PWGSC).

4. CAA Profile • This section provides an overview of the CAA under these headings: • Background; • Overview of Themes; • CAA Program Components; • Planned and Actual Expenditures by Theme; • Agenda Logic Model; and • Governance.

5. Background • In the 2006 Speech from the Throne, the Government committed to taking measures to achieve tangible improvements in our environment, including reductions in air pollutant and greenhouse gas (GHG) emissions. • Budget 2006 committed $1.7 billion, later increased to $2.2 billion, to the Clean Air Agenda (CAA) for the four-year period from 2007-08 to 2010-11. • The CAA comprises: • the Clean Air Regulatory Agenda (CARA), an integrated regulatory approach to reducing air pollutant (nitrogen oxides, sulphur oxides, volatile organic compounds, and particulate matter) and GHG emissions, and • a series of program measures to: • support action by governments, individuals, organizations and businesses in reducing emissions of GHGs and air pollutants, and • to provide effective responses to climate change. • The CAA is a horizontal initiative comprising eight themes and 45 programs. (Note: The Agenda was expanded to include the Clean Energy Dialogue as a component of the International Actions theme in 2009-2010). • A tabular overview of the eight themes is presented on the next page, followed by lists of the program components within each theme, identifying the department(s) that is delivering the program. • Detailed information on the Agenda and the program components are available in the 2008-09 Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives: http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481.

6. Overview of Themes Source: Environment Canada, 2008-2009 Departmental Performance Report, Supplementary Information, Table : Horizontal Initiatives. http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481. Unless otherwise stated, all budget allocation numbers reported in this document are from this source.

7. Clean Air Regulatory Agenda (EC) Industrial Sector Regulatory Actions (EC) Transportation Sector Regulatory Actions (EC, TC & NRCan) Consumer and Commercial Products Regulatory Actions (EC, NRCan) Indoor Air Quality Management Actions (HC) Science in Support of Regulatory Activities & Accountability (EC, HC) Emissions Reporting (EC) Emissions Trading System (EC) Enforcement and Reporting on Progress (EC) Policy Analysis (EC) Clean Energy (NRCan) ecoENERGY for Buildings and Housing (NRCan) ecoENERGY Retrofit (NRCan) ecoENERGY for Industry (NRCan) ecoENERGY for Renewable Power (NRCan) ecoENERGY Renewable Heat (NRCan) ecoENERGY for Technology (NRCan) ecoENERGY for Aboriginal and Northern Communities (INAC) Policy, Communications, Monitoring and Reporting (NRCan) International Actions (EC) International Obligations (EC, DFAIT) International Negotiations (EC, NRCan, DFAIT) Asia-Pacific Partnership (NRCan, EC, IC) PM Annex (EC) Clean Energy Dialogue (EC, NRCan, DFAIT) Partnerships (EC) 1. Community Partnerships (EC) Clean Transportation (TC) ecoMobility (TC) ecoTECHNOLOGY for Vehicles (TC) ecoENERGY for Personal Vehicles (NRCan) ecoENERGY for Fleets (NRCan) ecoFREIGHT Partnerships (TC) National Harmonization Initiative for the Trucking Industry (TC) Freight Technology Demonstration Fund (TC) Freight Technology Incentives (TC) Marine Shore Power Program (TC) ecoAUTO Rebate Program (TC, HRSDC) Vehicle Scrappage Program (EC) Analytical & Policy Support (TC) Indoor Air Quality (HC/NRC) Indoor Air R&D Initiative (NRC) Radon Strategy (HC) Adaptation (EC) Assist Northerners in Assessing Key Vulnerabilities and Opportunities (INAC) National Air Quality Health Index and Forecast Program (HC, EC) Climate Change and Health Adaptation in Northern / Inuit Communities (HC) Improved Climate Change Scenarios (EC) Innovative Risk Management Tools Regional Adaptation Action Partnerships (NRCan) Climate and Infectious Disease Alert and Response System to Protect the Health of Canadians (HC, PHAC) Management and Accountability (EC) Management & Accountability (EC) CAA Program Components 1

8. Planned and Actual Expenditures by Theme

9. Agenda Logic Model

10. Governance • EC is the lead department on the overall CAA and is responsible for the Clean Air Agenda-Results Management Secretariat (CAA-RMS). In all, eight departments (EC; NRCan; TC; NRC; INAC; HC; PHAC; and DFAIT) are responsible for themes and/or individual programs. Note: Two additional departments (Industry Canada and Human Resources and Skills Development Canada) are partners in specific programs. • The CAA accountability structure is composed of these interdepartmental committees: • DM Committee. • ADM Steering Committee • DG Theme Management Committees • DG Theme Leads Coordinating Committee (DGTLCC) • The DGTLCC is the primary operational committee responsible for implementing the elements of the CAA, comprising the leads from each of the eight DG Theme Management Committees. • The CAA-RMS is the main support structure for the CAA management and accountability theme. Its key functions include: • Providing support and coordination for the horizontal management of the CAA across departments; • Maintaining a database of financial resource information; • Analyzing financial and non-financial information to support an assessment of performance across the CAA, including achievement of results, use, allocation and sufficiency of resources, and risks; and • Providing relevant reports, as required, to the governance structure and TBS.

11. Methodology • This section discusses the plan for the CAA evaluation under these headings: • Scope and Coverage Limitations; • Approach and Methodology; • Theme and Program Evaluations Used for the Horizontal Roll-Up; • Evaluation Issues and Questions; and • Methodological Limitations.

12. Scope and Coverage Limitations Scope • The roll-up addresses the evaluation questions (page 15) set out in the 2008 horizontal evaluation plan. • It focuses on assessing theme and program level outputs and immediate outcomes (as reported in CAA program and theme evaluations) that are contributing to the achievement of the CAA intermediate outcomes (see Logic Model, page 9). The CAA long term outcomes were not assessed as part of this horizontal roll-up because they are expected to occur in a longer time frame. Coverage Limitations • The roll-up covers 39 (87%) of the 45 CAA program components (see page 14). • The combined four-year funding for the programs covered in this roll-up represents about 95% ($2.1 billion) of the $2.2 billion CAA allocation (see page 14). • Most of the contributing evaluations were conducted in 2009-10, thereby only covering 2 to 2½ years of implementation of the CAA activities. • Evaluations not included in the current roll-up will be reported separately when available. The current roll-up was finalized without the benefit of those missing components due to a commitment to provide this horizontal report in October 2010 to inform discussions on future CAA program activities.

13. Approach and Methodology Purpose • This report, which examines the relevance and performance of the Clean Air Agenda, is intended to meet reporting commitments and to contribute to ongoing clean air and climate change policy development. Audience • The CAA evaluation roll-up is intended to help meet the information needs of senior management at the departments and agencies that have developed and delivered CAA themes and program components, as well as at the Treasury Board Secretariat. Methodology • The sole line of evidence for this report is the content of the two thematic roll-up reports and 14 theme and program level evaluations available for review by September 23, 2010.2 This excludes evaluations that are still pending, as specified on page 14. • The following methods were used in the 14 theme and program evaluations that contributed information to this roll-up: • key informant interviews (14), • literature review (6), • document review (14), • analysis of performance data (5), • analysis of secondary data (1), • case study (5), • survey (2), and • experts group (1). • The evidence was extracted from the reports to produce crosswalk matrixes that mapped equivalent data from multiple theme and program evaluations to fields defined by the CAA evaluation questions.

14. Theme and Program Evaluations Used for Horizontal Roll-UpLegend: ✓-Evaluation of this theme/program was included in this roll-up; X – Evaluation of this theme/program is in progress or planned. a - This total does not include additional funds under a reprofiling of CARA that was approved in 2009. b - This program was not subject to a discrete evaluation as per NRCan’s evaluation plan. c - These two programs extended their Terms and Conditions for one year and will thus expire in 2011-2012. As a result, the evaluation was pushed back until winter 2011.

15. Evaluation Issues & Questions Relevance – Are CAA activities consistent with and do they contribute to addressing federal government clean air priorities and the key environmental needs of Canadians? • Are CAA activities aligned with federal government priorities? • Are CAA activities connected with key environmental needs? Success – Has the CAA achieved its intended outcomes? • To what extent has the CAA achieved its intended outcomes? Cost-Effectiveness – Are the most cost-effective and efficient means being used to achieve outcomes? • Are there more cost-effective and efficient means of achieving CAA objectives? • How could the efficiency of CAA activities be improved? Design & Delivery – Is the CAA designed and delivered in the best possible way? • Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time? • Is the CAA management and accountability structure in place and functioning as anticipated? • Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers? • What are the best practices and lessons learned from CAA activities?

16. Methodological Limitations Limitations affecting the horizontal roll-up • Roll-up vs. horizontal evaluation: The large scale of CAA activities (i.e., $2.2B, 45 programs and 8 departments) dictated the conduct of several theme and program level evaluations, and a subsequent horizontal roll-up of these evaluation findings, rather than an umbrella evaluation of the overall CAA. • Program maturity: Most evaluation evidence was collected during early program implementation and therefore relies on limited performance data and/or data focusing on outputs rather than outcomes. Annual performance reports were however developed for the first two years of the CAA. • Lines of evidence: Many of the evaluations contributing to this report were heavily reliant on information drawn from key informant interviews. The roll-up of the interview evidence cannot be considered highly reliable given the open-ended nature of the interviews, the variations in questions, and the self-interest of a majority of the respondents (about 46% of the 547 interviews conducted were with federal government employees managing and/or delivering the programs). • Level of Focus: It is difficult to draw conclusions about some of the CAA evaluation questions because of the level of detail and focus of the contributing evaluations. Specifically, theme-level evaluations and theme-level roll-ups tend to produce generalized or heavily qualified findings that cannot be quantified or consolidated. Underlying Limitations The following methodological and analytical limitations were reported by more than one of the theme evaluations. (The number in parentheses indicates the number of evaluations identifying the limitation) : • Program maturity (4): The programs were not completely implemented and it was too early to assess outcomes. • Lines of evidence (3): Only two or three lines of evidence were used. • Limited performance data (6): The available data were limited to output/activity level indicators. • Policy processes (5): It was difficult to access/collect relevant output and outcome data on such processes. • Cost data (4): Current and/or sufficiently detailed program and activity level financial data was not available.

17. Findings This section rolls-up the findings and conclusions from the program and theme-level evaluations that are relevant to each horizontal-level evaluation question.

18. Relevance1. Are CAA activities aligned with federal government priorities? Evidence • Overall CAA activities and outcomes are seen to align with federal priorities as expressed in documents and statements such as: • Budgets and speeches from the Throne; • Turning the Corner, the April 2007 Government of Canada plan to reduce greenhouse gases and air pollution; • UNFCCC commitments, including the commitment under the Copenhagen Accord to a 17% reduction in GHG emissions from 2005 levels; • Canada’s Performance 2008–09 that identifies “a clean and healthy environment” and “healthy Canadians” as two of the government’s thirteen outcome areas; • A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act, May 2010. • The evaluations confirmed that the program components they reported on are aligned with departmental priorities and program activity architectures. Finding • CAA activities are aligned with federal priorities.

19. Relevance2. Are CAA activities connected with key environmental needs? Evidence • The CAA evaluation plan listed three indicators for this question: demonstration of the utility/rationale for CAA activities; CAA programs remain relevant as new scientific evidence on air quality and climate change emerges; and, evidence that CAA program architecture has adapted to new evidence and changes of government policy and priorities. • The theme and program evaluations, with the exception of the M&A theme evaluation, considered “needs” in the course of assessing the relevance of programs and activities. However, the interpretation of the concept and the kinds of evidence gathered varied. Some looked at key informant opinions about workloads or the responses of beneficiaries and stakeholders as evidence of need, while others referred to research results and/or secondary data sources to document air pollutant and GHG emission levels. The Clean Energy theme report demonstrated a continued need for several of the theme’s program activities. • Three of the thematic evaluations examined whether new scientific evidence had emerged that would raise questions about the relevance of CAA activities – none was found. • There were no discussions or reports of modified program architecture. Finding • Several of the theme and program evaluations provided evidence to demonstrate that CAA activities are connected to key environmental needs.

20. Success3. To what extent has the CAA achieved its intended outcomes? Approach • The roll-up assessed the extent to which each CAA intermediate outcome (see Logic Model, page 9) has been achieved by examining the “success” evidence from the evaluations for the themes that are expected to be the major contributors to each outcome, as identified below:

21. Success3. To what extent has the CAA achieved its intended outcomes? Approach (continued) • For each of the CAA intermediate outcomes, success information is presented under three headings: • The plans made to achieve the outcome, including which themes/programs were involved and the nature of interventions -- incentives; regulations and agreements; or capacity building -- that are being used. • The term “capacity building” is used here to refer to activities that involve obtaining or distributing information with the goal of changing individual and organizational behavior including research, public information products and services, social marketing campaigns, training and educational activities, and demonstration projects. • The evidence from the theme and program evaluations about the extent to which planned outputs have been produced and intended outcomes achieved. • Findings that can be drawn from the available evidence. It is important to note here that outcome data were not available for most of theme and program level evaluations because programs had not been fully implemented when evaluated, or their outputs were only expected to have the impacts that will contribute to key outcomes some months or years later. The findings sections that follow do not repeat this qualification, but rather highlight where outcomes data are available. continued......

22. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved Success3. To what extent has the CAA achieved its intended outcomes? Estimate of Progress • An indicator (see example below) was developed to provide a high level estimation of progress made toward the achievement of each CAA intermediate outcome based on the evidence reported by the theme and program evaluations reviewed: • No outputs – no outputs because program implementation was delayed or terminated. • Partial outputs – outputs are being produced but not to planned levels, or work has been significantly delayed. • Outputs delivered – most planned outputs are being delivered, but insufficient time has elapsed for their contribution to CAA intermediate outcomes to have been experienced. • Outcomes likely achieved – given the extent of outputs delivered and program maturity (i.e., sufficient time has elapsed since program implementation), and assuming that the CAA results logic is accurate, it is likely that the intended intermediate outcome is being achieved. There was however no data collected in the program or theme evaluation to assess whether the outcome was being achieved. • Outcomes achieved – there is reliable evidence that the intermediate CAA outcome has been achieved.

23. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Canadians take actions in response to forecasted levels of air quality. The Plan • The seven Adaptation theme programs were designed to improve Canada’s capacity to reduce health risks from and adapt to a changing climate and air pollution. The program designers expected increased capacity to be achieved through research and the development of tools, partnerships and standards. Research and tool development is being done directly by the participating departments and by other organizations supported through contribution funding. • The theme targets: the North; human health; and infrastructure. Evidence • The Adaptation programs, with one exception, were launched in 2008-2009, not in 2007-2008, the first year of the CAA. The exception is the AQHI which is the extension of a predecessor program. • The theme evaluation, drawing on four program evaluations, reported that the programs were being implemented as planned, although two of them were at risk of partial or delayed implementation. • The available evidence showed that where implementation has begun, the programs are beginning to achieve immediate theme-level outcomes: increased capacity to conduct science; greater collaboration; and increased availability of and access to products. Findings • There is some preliminary, although weak, evidence that the Adaptation theme is making progress toward achieving the theme’s immediate outcomes that are expected to contribute directly to this CAA intermediate outcome.

24. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:International air quality agreements are consistent with Canada’s interests The Plan • The PM Annex component of the International Actions theme is working toward the negotiation of an annex to the 1991 Air Quality Agreement. Under such an agreement, the U.S. and Canada would commit to reducing emissions of PM (particulate matter) and its precursors. PM comprises air-bound, microscopic pollutants that have negative impacts on air quality, the environment and human health. Evidence • The PM Annex initiative has two major outputs: i) partnerships and processes; and ii) formal and informal consultations with domestic stakeholders. • The International Actions theme evaluation reports evidence that some progress was made toward developing partnerships and processes with the U.S. to establish the PM Annex. • The evaluation found no evidence that stakeholders were consulted directly and it was not clear how Canada’s position was developed or how the views of domestic partners and stakeholders have had an influence or been meaningfully reflected. • Changes in policy direction and positions on key issues in both Canada and the US have resulted in delays in achieving the PM Annex. Findings • PM Annex activities have produced outputs related to partnerships and processes but there is little evidence to show that consultations with domestic stakeholders have helped to inform the development of Canada’s position in PM Annex negotiations. Such consultations are necessary to ensure the PM Annex is consistent with Canada’s interests.

25. The Plan The CAA expected to achieve this intermediate outcome through: the Indoor Air Quality theme, which is focusing on capacity building activities including research on improved ventilation, distribution and other indoor air quality solutions, construction of a indoor air research laboratory, creation of a national committee on indoor air quality; the mapping of radon zones; testing radon levels at federal sites and a radon public awareness campaign; and The CARA theme that planned new federal guidelines (regulations) for indoor air contaminants, and a national radon risk management strategy. Evidence Because it was done during the third year of a four-year program and many of the outcomes are not anticipated until the end of the four-year period, the IAQ evaluation examined the progress that had been made under the theme’s two program components. It found that they were generally on track to deliver their planned outputs. The notable exceptions are: the programs have not delivered the planned radon public awareness campaign or the assessments of the indoor air quality public awareness effort due to restrictions on the conduct of public opinion research; and, radon testing of federal buildings is behind schedule. The CARA evaluation reported that federal guidelines for indoor radon, carbon monoxide and ozone levels have been published within planned time frames and that related work is generally on track. Findings The Indoor Air Quality Theme and CARA have produced, or are on track to produce, the outputs that are expected to contribute to the achievement of theme and CAA intermediate outcomes, but it is too early to expect that the outputs have had their intended impact. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Reduction measures for indoor air pollutants and radon exposure have been implemented

26. The Plan The Industrial Regulations component of the CARA theme was designed to develop and implement sector-specific GHG emissions regulations. Evidence The CARA evaluation reports that the development of regulations for industrial greenhouse gas emissions, as well as the development of emissions trading and offset systems, have been affected by “directional change in policy.” The program, however, has undertaken many of the planned activities (consultations, research, drafting instructions), but further work has been deferred. Findings CARA has not yet had the opportunity to produce the outputs needed to achieve the GHG and air pollutant emission reductions required for the achievement of this CAA outcome. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency

27. The Plan The Clean Energy theme offered homeowners, as well as small and medium –size organizations, incentives to retrofit or construct energy efficient homes and buildings through two programs: ecoENERGY Retrofit ($805 M) and ecoENERGY for Buildings and Houses ($61 M). Evidence At the end of its first two years (2007-2008 and 2008-2009), the ecoENERGY Retrofit for Homes program had already achieved 64-80% of its targeted range for GHG emission reductions and energy savings. Evidence suggests that 84% of energy savings are directly attributable to the Program. Moreover, homeowners are undertaking double the number of retrofits they had originally planned and 60% are motivated to undertake further measures outside of the Program. At the end of its first two years (2007-2008 and 2008-2009), the Buildings and Houses program had achieved 63-71% of its targeted range for energy savings and 80% of its GHG emission reductions and had effectively extended the reach of NRCan’s energy efficiency programs in the residential sector. The program has also exceeded its target of 5,000 trained energy advisor professionals in the construction of energy efficient new homes and buildings. The ecoENERGY Retrofit for Small and Medium Organizations program had achieved GHG emission reductions and energy savings but not to the extent anticipated when targets for the programs were set. Furthermore, attribution of this progress to the program’s financial incentive activities is only partial as some firms would have implemented these changes on their own. For the most part, capacity building activities (e.g., training and design validation), are increasing the capacity of commercial / institutional energy users to adopt energy management best practices and implement energy efficiency measures. In turn, this contributes to the outcome of energy savings and GHG emission reductions Findings Evidence shows that, although they have not yet fully reached all of their intended targets, Clean Energy programs are achieving GHG reductions and energy efficiency improvements in the Canadian homes and buildings sector. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings

28. The Plan Two themes are contributing to the goals of increasing fuel efficiency and reducing GHG and air pollutant emissions within the personal and commercial transportation sectors: The CARA theme is focused on developing and implementing a range of regulations and related products; and the Clean Transportation theme comprises capacity building and incentive programs intended to increase the adoption of fuel efficient transportation technologies and behaviours, as well as plans to establish voluntary emission reduction agreements with the air, railway and marine sectors of the transportation industry. Evidence The CARA theme evaluation found that the development of transportation-related regulations was happening as planned and within scheduled time frames. The Clean Transportation theme evaluation found that, with small exceptions and some delays, the initiative was delivering its planned activities and outputs. With respect to outcomes, it found evidence that the ecoAUTO Rebate program , an incentive program, had contributed to an increased uptake of fuel-efficient technologies in the personal vehicle market. There was also evidence of some increased awareness of energy efficient transportation technologies among target audiences. With respect to the capacity building programs, both the theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found the required outputs had been delivered but that outcomes could not be assessed due to a lack of baseline and outcomes data and the challenges of attributing changes in consumer and business behaviours to specific interventions. The Clean Transportation theme evaluation reports that the voluntary agreements with the air and railway sectors have been renewed or maintained, but a similar agreement for the marine sector has not been developed. Findings By and large, the CAA transportation-related activities have delivered the regulatory, incentive and capacity building outputs that are expected to contribute to the achievement of this CAA outcome. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Emissions of GHGs, air pollutants, energy consumption have been reduced from modes of transportation

29. The Plan It was expected that this outcome would be realized as a result of capacity building outputs such as public information services, activities promoting smarter energy use, support for the development and deployment of clean energy technologies, and fuel efficient driver training delivered by the Clean Energy and Clean Transportation themes, as well as regulatory actions developed through the CARA theme. Evidence The Clean Energy evaluation reports that awareness building activities and adoption of technologies under the ecoEnergy for Industry program contributed to reduced emissions. At the time of the evaluation, the program was within its target range for energy savings and GHG reductions. It however reports that, while the activities and programs funded within the ecoEnergy for Technology Initiative lay the groundwork for increased deployment of clean power generation, reduced emissions from this program will depend on the degree to which these technologies are deployed on a large scale. The Clean Transportation theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found that the planned capacity building outputs were being produced. The theme evaluation also highlighted the challenges of attributing changes in consumer and business behaviour to specific capacity building interventions. With respect to regulatory interventions, the CARA theme evaluation reported that the planned development of energy -efficiency standards for consumer products and regulations to reduce the emissions of volatile organic compounds (VOC) in the manufacture and use of consumer and commercial products were on track, but that the impact of the regulations would not be measurable until some years after they had come into effect. NRCan’s evaluation of Equipment programming indicated that its Energy Efficiency Standards and Labelling Program contributes to increasing the minimum energy efficiency of equipment manufactured by suppliers, and therefore purchased by consumers, thereby contributing to reductions in energy use and associated GHG emissions. Findings There is evidence that the Clean Energy, Clean Transportation and CARA themes have produced the regulatory and capacity building outputs designed to increase the use of efficient and safe practices and products and some evidence that this outcome is being achieved as a result. It is however too early to conclude on the achievement of this outcome for 2 of the 3 themes involved. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products

30. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources The Plan • The CAA expects to achieve this outcome through the incentives delivered by two Clean Energy theme programs: • ecoENERGY for Renewable Power ($276.0 M) provides incentives for the producers of renewable power (e.g., from wind, low impact hydro, biomass and solar photovoltaic); and • ecoENERGY for Renewable Heat ($36.0 M) uses contributions to commercial, industrial and institutional users to encourage the installation of solar water and air heating systems. As more are installed, prices are expected to fall and the consumption of conventional energy should decline. Evidence • By April 2009, Renewable Power projects with a total capacity of 3,100 megawatts had been commissioned or were under construction – the program’s target is approximately 4,000 megawatts by March 31, 2011. • During its first two years, the Renewable Heat Program supported 334 solar thermal projects – its target is 700 systems by March 31, 2011. Findings • NRCan’s Renewable Energy Evaluation concluded that, after two years, the renewable energy programs have delivered planned outputs and are on track to reach most of their intended outcomes. In addition, it expressed the opinion that many of the programs’ long-term outcomes will be achieved, although it is not possible to directly attribute the achievement of these long-term outcomes to the programs due to interacting factors and considerations.

31. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:Canadian communities and user groups use tools and information to assess climate change risks and plan adaption strategies. The Plan • Several of the Adaptation Theme programs included capacity development activities (i.e., the development and distribution of tools and information such as guidelines, methodologies, data tables, models, and web pages) to help researchers, government departments and agencies, businesses, community groups and others assess risks and plan adaptive measures. Evidence • The Adaptation Theme evaluation reported that the evaluations completed for four of the seven programs indicate that the programs were generally being implemented as planned and that Improved Climate Change Scenarios may not be fully implemented within the program timeframe. • Available information indicates that the use of adaptation information and products is most likely to be used by groups who have an immediate interest in climate change impacts (e.g., scientists or individuals who are at-risk from adverse health impacts). Findings • The evidence points to increased use of tools and information by groups with a clear interest in, or an incentive to use, adaptation products and information.

32. No outputs Partial outputs Outputs delivered Outcomes likely Outcomes achieved CAA Intermediate Outcome:International climate change agreements are consistent with Canada’s interests The Plan • The International Partnerships and Negotiations component of the International Actions Theme is the most important contributor to the achievement of this CAA outcome. Its activities are intended to ensure that Canadian environmental and economic interests are advanced through international negotiations. Note: For the purposes of evaluation, “Canada’s interests” are those goals and interests defined by the Government of Canada in federal negotiating instructions and formal policy statements. Evidence • The International Actions evaluation reported that the International Partnerships and Negotiations program had produced planned outputs. • The evaluation concluded that Canada’s interests are reflected in international climate change agreements, although it is difficult to attribute this directly to the activities under the International Actions theme. As well, there is clear evidence of enhanced consultation activities but they have not necessarily resulted in increased consideration of stakeholder interests in Canada’s positions Findings • There is evidence that the International Actions theme has, at least partially, achieved this CAA intermediate outcome.

33. SuccessProgress toward Outcomes: Summary

34. Cost-Effectiveness/ Alternatives4. Are there more cost-effective and efficient means of achieving CAA objectives? Evidence • Opinions about alternative more efficient approaches: Key informants were either satisfied that the current program models were as or more efficient than alternatives, or they did not know of any realistic alternatives. • Comparison of CAA activities to other comparable programs: Only the Clean Transportation Theme evaluation reported the results of a comparative analysis. It found that comparable programs exist for most Clean Transportation programs, but the availability of cost information for the comparable programs limited the analysis to the ecoAUTO Rebate and the Vehicle Scrappage programs. The Vehicle Scrappage program seems to be more efficient than United States and United Kingdom initiatives based on calculations of incentive cost per vehicle scrapped and total cost per vehicle scrapped. A comparison of the ecoAUTO Rebate program and the U.S. Cash for Clunkers program shows similar unit costs for administration. 3 • Opinions about whether CAA investments are a good use of public funds: While the evaluations do include key informant suggestions for program refinements and adjustments, the overall tenor of the evidence is that program staff and other stakeholders are generally satisfied that the programs are doing their job at reasonable cost and that no realistic alternatives are available. • Costing emission reductions: The Clean Transportation evaluation calculated a cost per tonne of GHG emissions reduced based on forecasted GHG emissions reductions and original program budgets. The results suggest that the Vehicle Scrappage and ecoAUTO Rebate programs are not cost-effective. Finding • While there is key informant and comparative evidence that CAA activities are efficient, calculations of the cost per tonne of GHG emission reductions suggest that some programs, particularly incentive programs, are not cost-effective.

35. Cost-Effectiveness/ Alternatives5. How could the efficiency of CAA activities be improved? Evidence • The evaluations sought key informant opinions about ways to improve the efficiency of the programs they were associated with. • The suggestions that were offered and reported tend to be specific to particular programs, even to particular activities. • A comparison of the evaluations failed to identify any suggestions that were repeated across themes and programs. • During the first two years of operation CAA theme actual expenditures were under budget (see page 8). Finding • There was no evidence of efficiency improvements that would be relevant to CAA activities generally.

36. Design and Delivery6. Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time? Evidence • Information about the implementation status of 40 of the 45 CAA programs is available. • The majority of the programs have been implemented as planned, that is they are producing the expected outputs and, generally, within the expected timeframes: • One program, Clean Transportation’s ecoAUTO Rebate program (TC) has been completed; and • Twenty-nine programs were identified as on track – delivering their outputs as planned. • One (1) CAA program was not implemented as planned • i.e. Community Partnerships (EC). • Work on two CARA initiatives – the Emissions Trading System (EC) and the Industrial Sector Regulatory Actions (EC) – has been deferred. • Seven programs have experienced challenges resulting in delayed or partial implementation: • The Clean Transportation programs ecoMobility (TC); Marine Shore Power Program (TC); National Harmonization Initiative for Trucking Industry (TC), and ecoFREIGHT Partnerships on Freight (TC); • The International Actions initiatives PM Annex (EC) and Asia-Pacific Partnership (NRCan); and • The Radon Strategy (HC). Finding • The majority (30 of 40) of the CAA programs evaluated have been implemented as planned, six have been implemented with significant schedule and/ or output changes, two were not implemented and two are described as deferred.

37. Design and Delivery7. Is the CAA management and accountability structure in place and functioning as anticipated? Evidence • The CAA management and accountability structure is the subject of a separate evaluation that concluded, in part, that: • The M&A Theme has brought coherence to the horizontal reporting of CAA activities and results and has emerged as a strong success story, in large part because of the CAA–RMS. Central agency representatives said that it was a key model to follow for reporting on horizontal initiatives. • Roles and responsibilities for CAA management have been clearly defined, processes to support decision making and coordination have been established, and results are being reported to Parliament and Canadians in a transparent manner. • There was limited evidence, however, of policy and program adjustments being made within the CAA horizontal decision-making structures. When needed, horizontal collaboration has occurred through other more flexible channels. • The general tenor of the management and accountability evaluation findings at the theme level tend to validate the M&A theme evaluation conclusion: horizontal management structures have been implemented and play an important role in reporting, but have limited roles in program and policy decision-making, in part due to limited interdependencies between themes (see page 20). Instead, decision making operates on vertical lines of accountability and, where there are interdependencies between programs, bilateral planning and coordination solutions are used. Finding • The agenda and theme level horizontal management structures have been implemented as planned. Their functioning in terms of CAA reporting responsibilities exceeds expectations, but they have not played any significant role in decision-making.

38. Design and Delivery8. Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers? Evidence • The CAA evaluation plan listed three indicators for this question: • Populated performance data systems: The key informant and document review evidence reported by the theme and program level evaluations suggests that the programs are meeting basic departmental and horizontal reporting requirements. • The evaluations do not contain enough specific information to do a program-by-program analysis, but taken together, their observations suggest that a minority of programs have fully functioning systems. • The most commonly identified issues were: the lack of an adequately structured and maintained system; processes or systems that focus on activities and outputs, but lack key outcome data; and indicators that are not relevant or appropriate. • Six (6) of the nine (9) that examined performance data questions made recommendations for improvements to performance data. • Decisions based on performance information: Only two evaluations reported on the use of performance data in decision making: the International Actions evaluation observed that it was used for departmental reporting, but little else, while the Clean Transportation document review and key informant interviews found examples of the use of such data in program management decision making, as well as reporting. • Baseline data: Only a few programs have collected or will be able to collect baseline data. As a result it will be difficult, if not impossible, to assess the achievement of intermediate and longer-term outcomes whether the outputs involved incentives, capacity building activities or regulations. Finding • It appears that CAA programs have established systems or processes to gather and maintain activity and output data that supports program management, but some are not collecting the baseline and outcomes data needed to measure and evaluate performance.

39. Design and Delivery9. What are the best practices and lessons learned from CAA activities? Evidence • The following best practices and lessons learned are those at the Agenda level that cut across several CAA themes or programs. Other best practices and lessons learned can be found in the CAA theme and program evaluation reports. • Best Practices: A best practice is a methodology or work process that has been demonstrated to achieve superior results. The evaluations reported a number of best practices, but often fail to provided evidence to demonstrate that the activity described was a superior solution. • The best practices at the Agenda level reported by the Management and Accountability evaluation were: • Requiring approval of CAA performance reports by DMs of partner departments and agencies before the reports are submitted to TBS. • The detailed charter, and the CAA–RMS implementation of the terms of the charter. • The approach of the CAA–RMS to ensure effective financial and performance data collection, compilation and reporting (acknowledged by TBS to be a model of reporting on a horizontal initiative). • Lessons Learned: A lesson learned describes an aspect of a project or program that did or did not work well, and identifies the risk of ignoring the lesson. In all, six of the evaluations reported a total 19 “lessons learned.” • The evaluations tended to provide only a brief description of the lesson without context or identification of risk. • While the lessons tended to be specific to a particular program or activity, it is worth noting that four of the evaluations (CARA, Adaptation, Renewable Energy and IAQ) listed nine lessons learned that called for proactive communication to and engagement of external stakeholders, including communities and individuals. Finding • The CAA–RMS led processes for financial and performance data compilation and reporting represents a best practice for horizontal initiatives.

40. Overall Observations • This section presents observations in two groups: • Program observations: Observations related to overall CAA relevance and performance • Evaluation observations: Observations related to technical aspects of horizontal evaluations

41. Program Observations Relevance • The CAA programs are aligned with government priorities and are connected to key environmental needs. Achievement of Outcomes • Early evidence shows that a large majority of the planned activities are being implemented and delivered. • As acknowledged in the 2008 CAA horizontal evaluation plan and the theme and program evaluations, the impacts of most of the CAA programs will not be felt or measureable for some time, even if the program has been fully implemented. • Some outcome data for incentives programs (ecoEnergy for Retrofit, Renewable Energy, eco AUTO, and Vehicle Scrappage) suggests that there is increasing uptake of energy efficiency products and services, fuel-efficient technologies and clean energy sources, but there are also outstanding attribution challenges. • With the exception of Clean Energy programs, there is mostly anecdotal evidence about the likely impacts of CAA capacity building activities and, without baseline data and given restrictions on survey research, it will be difficult to reliably assess those impacts in the future. • Literature reviews have found international evidence about the effectiveness of regulatory interventions, but evaluations of the Clean Air regulations have not yet been done. Cost Effectiveness • There is preliminary evidence that incentive programs, like the ecoAUTO Rebate program and the Vehicle Scrappage program, may not be cost-effective when measured in terms of estimated emission reduction. Design and Delivery • The CAA horizontal management structures have enabled very successful horizontal reporting but have had limited usefulness for program and policy decision-making in light of the limited interdependencies between CAA themes and the maintenance of vertical lines of departmental accountability.

42. Evaluation Observations Horizontal Evaluation Frameworks • The development of the horizontal evaluation plan provided some degree of consistency across the theme and program level evaluations, but not enough to facilitate a strong roll-up of results. Future horizontal evaluation planning might benefit from establishing requirements for all theme and program evaluations to address a small number of core questions, with well defined indicators and comparable/equivalent data sources. Performance Measurement • The majority of the theme and program evaluations reported that they encountered problems because there was limited outcome data. In many cases this was because the program had not reached the stage of its implementation where outcomes could be measured, or because of methodological issues such as limitations on public opinion research. In other cases, however, it may be that program managers did not collect and maintain outcome data. • Evaluation planning would benefit from scheduling early reviews of program performance measurement frameworks and their implementation. Common Terminology • The roll-up of the theme and program evaluation results was complicated by different interpretations of some concepts and terms such as needs, utility/rationale, unintended outcome, best practice, and lessons learned. While variation in interpretation and understanding of these terms is common and not usually an issue with a single evaluation, a horizontal evaluation plan would benefit from including a terminology lexicon to obtain consistent evidence across several evaluation units and numerous evaluators.

43. Endnotes • Descriptions of each of the CAA program components are included in the Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives. The 2008-09 supplementary table is published at http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481. • The primary information sources for this evaluation roll-up were the following program and theme-level evaluation reports: CARA: Evaluation of the Clean Air Regulatory Agenda -DRAFT (June 14, 2010), Audit and Evaluation Branch, Environment Canada. CARA: Evaluation of the Equipment Sub-Sub Activity - Draft Final Report (June 28, 2010), NRCan Strategic Evaluation Division. (Note: This evaluation covers NRCan’s Energy Efficiency and Standards Labelling Program, which is a sub-component of CARA under the Consumer and Commercial Products Regulatory Actions program). Clean Energy: NRCan’s Clean Energy Thematic Report - Draft (September 2010), NRCan, Strategic Evaluation Division. Clean Energy: Renewable Energy Evaluation – Final Report (July 2010), NRCan, Strategic Evaluation Division. Clean Energy: Evaluation of the Clean Electrical Power Generation (CEPG) S&T Sub-sub Activity (PAA 2.1.4.5) - Final Evaluation Report. (September 17, 2010),NRCan, Strategic Evaluation Division. Clean Energy: Evaluation of the Oil and Gas Sub-sub Activity. Draft Report (September 22, 2010), NRCan, Strategic Evaluation Division. Clean Energy: Evaluation of Energy Efficiency for Industry, Housing and Buildings – DRAFT Report (September 15, 2010), NRCan, Strategic Evaluation Division. Clean Transportation: Government Consulting Services, Report on the Evaluation of the Clean Transportation Theme of the Clean Air Agenda (July 2010), Transport Canada. (Note: This report covers the Clean Transportation theme programs delivered by Transport Canada and Environment Canada). Clean Transportation: Evaluation of the Transportation Sub-Sub Activity - Final Report (July 6, 2010), NRCan, Strategic Evaluation Division. (Note: This evaluation covers NRCan’s participation the transportation sector regulatory actions and six programs, two of which are part of the Clean Transportation theme). Indoor Air Quality: Government Consulting Services, Evaluation of the Indoor Air Quality Theme of the Clean Air Agenda (June 2010), Health Canada and National Research Council. Adaptation: Evaluation of the Clean Air Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings -DRAFT (July, 2010). Audit and Evaluation Branch, Environment Canada. Adaptation: Evaluation of the National Air Quality Health Index Program (February 2010), Audit and Evaluation Branch, Environment Canada. Adaptation: Evaluation of the Improved Climate Change Scenarios Program - Final Draft Report (June 2010), Audit and Evaluation Branch, Environment Canada. International Actions: Evaluation of the International Actions Theme of the Clean Air Agenda - Draft Final Report (June 2010), Audit and Evaluation Branch, Environment Canada. International Actions: Marbek | Stratos. Evaluation of the Canada-U.S. Clean Energy Dialogue - Draft Report, Audit and Evaluation Branch, Environment Canada. Management and Accountability: Evaluation of the Clean Air Agenda Management and Accountability Theme - Draft (Revised July 13, 2010), Audit and Evaluation Branch, Environment Canada.

44. Endnotes • It should be noted, though, that the United Kingdom and United States programs were adopted for economic stimulus reasons rather than to reduce transportation emissions. • The NRCan Evaluation of the Equipment Sub-Sub Activity (contributing to the CARA Theme) made similar calculations for the two non-CAA programs it reported on (equivalent data for the CAA programs was not available): (page 42) • The Accelerated Standards Action Program was expected to reduce GHG emissions by 2.8 Mt/year in 2010. In total, $28.4 million was spent under the Accelerated Standards Action Program. Therefore, if the Accelerated Standards Action Program achieved its GHG emissions reductions estimate, it will result in $10.14 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010. • EnerGuide for Industry was expected to reduce GHG emissions by 1.4 Mt/year in 2010. In total, $2 million was spent under EnerGuide for Industry. Therefore, if EnerGuide for Industry achieved its GHG emissions reductions estimate, it will result in $1.43 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.