2012 EEEN Forum 9-10 February , Leuven , Belgium Paula Kivimaa a & Vivi Niemenmaa b

1. How to evaluateclimatepolicy: Case of an evaluation-basedaudit of Finnishclimate and energystrategy 2012 EEEN Forum 9-10 February, Leuven, Belgium Paula Kivimaaa & Vivi Niemenmaab a Senior Researcher, FinnishEnvironment Institute b PrincipalPerformanceAuditor, National Audit Office of Finland

2. Contents • Challenges for evaluatingclimatepolicy • Performanceaudit in comparison to policyevaluation • Evaluation-basedperformanceaudit of Finnishclimate and energystrategy • Conclusions on challenges Paula Kivimaa, SYKE

3. Challenges for evaluatingclimatepolicy • Uncertain and long-termcause-effectchains • Emergence of new scientificknowledge • Finalimpactmeasurementnotpossible • Multipleothercauses and policiesinfluencing in desired and oppositedirections • E.g. economicfluctuations, policiessupportingemployeemobility, municipalinterests • Horizontalinvolvement of severaladministrativesectors and policydomains an important feature of climatepolicy • Howdowedefineclimatepolicy? • Impacts on effectivess and evaluation • Slowlychangingpractices • Despiteawareness, it is difficult to change daily practices, administrativeprocesses, etc. • Marginalchangeor no change? Cause for change? Paula Kivimaa, SYKE

4. Findingsfrom a climatepolicymeta-evaluation • Therearenotenoughsystematicevaluations of climatepolicy to supportevaluation-basedpolicymaking (Haug et al. 2010) • Existence of bothtechnical-analytical and reflexiveevaluationscarried out by a number of actors (Huitema et al. 2011) • “Several long-standing measures reported as climate policies were initially designed as responses to other problems” and “climate policy evaluations do not, in general, challenge dominant framings of the policy problem as one of market and state failure with regard to greenhousegasemissions” (Haug et al. 2010) • Evaluationsshould (1) address the complexity of climatepolicymaking, (2) bereflexive in challengingpolicygoals and means, and (3) beparticipatory (Huitema et al. 2011) Paula Kivimaa, SYKE

5. Performanceaudits • ”the independentexamination of the efficiency and effectivenss of governmentorganizations, operations, orpolicies, withdueregard to the economy’”(INTOSAI, 2010:11) • Startsfrom the rationalistic side in thatauditsarecarried out from the viewpoint of whatstatescando and shoulddobetter and relates to performance-based management • Notsomuchconcernedwithe.g. rational social choiceorfuturepaths • Oftenhave a narrowerfocusthanevaluation as a whole • focused on ex-post and have to bejustifiedbydirectorindirectlinks to governmenteconomy • Based on principles of goodgovernance and decisions and goals set up in legislation (van der Knaap, 2011) • Carried out bysupremeauditinstitutions Paula Kivimaa, SYKE

6. Case of evaluation-basedaudit of Finnishclimate and energystrategy • Carried out between August 2010 and November 2011 (circa 14 effectiveworkingmonths) • Focus on the quality of policypreparation & the consistency, effectiveness and cost-efficiency of implementation • Addressedcomplexitythroughmultiplemethods and data sources & addressedparticipationthrough a stakeholderquestionnaire, interviews, and a focusgroup • Wasnotreflexive in a sensethat the main goalsweretaken as givenfrom the EU level and didnotaddress the framing of goals and measures Paula Kivimaa, SYKE

7. The effectivenessmodel (source: Performance Audit Manual of the NAOF) Feedback to objectivesetting Paula Kivimaa, SYKE Main focus of climate and energystrategyaudit

8. Audit questions 17.10.2011 • Was the preparation of the strategy of goodquality? • Was the preparationorganisedeffectively, crossingadministrativesectors, transparently and involvingdifferentstakeholders? • Was the informationbaseversatile and sufficient, and was the governmenteconomycomprehensivelyacknowledged? • Howwerepolicyaims, instruments and theirinterrelationsconsidered? • Howwere the implementation and responsibilitiesplanned? • Has the implementation of the strategystrived for consistency, effectiveness and cost-efficiencyfrom the perspective of climatechangemitigation? • Howdoesclimatechangemitigationshows in the preparation of governmentbudgets? • Is implementationmanagedeffectively and withsufficientresources? • Hasimplementationbeen in linewith the strategystatements? • Hasmonitoring and evaluationbeenorganisedappropriately? • And moretentativeevaluation of impacts and effectiveness Paula Kivimaa, SYKE Paula Kivimaa, SYKE 8

9. Audit criteria (moretentativelyeffectiveness) Paula Kivimaa, SYKE

10. Methods and material Paula Kivimaa, SYKE

11. General view of implementation in differentsectors, questionnaire (5=excellent, 1=poor)

12. Examplefrom the transport sector • Policies (resources, measures) • Sectoralobjectivesetting (options, informationbase) Paula Kivimaa, SYKE Adaptedfrom Monni & Raes (2008)

13. Measuresinfluencing transport emissions

14. Effectiveness and emissionstentatively

15. Vehicleefficiency: kilometres per tonne of CO2

16. Summary of implementation • Significant new measures in renewableenergy, transport and energyefficiency of buildings • Major flaws: • Agriculture, communitystructure and landuse (thataffectpossibilities to reducetraffic and itsemissions) • No real and innovativeopenings to promotemicro-orconsumer-generation of renewableenergy • Budgetfunds for the transport sectoraresmall • Monitoring and follow-up • The officialmonitoring of strategy is weak, butfulfilledpartlythroughmonitoring of governmentprogrammes and to reporting to the UNFCCC and EU, which in turnarecarried out well.

17. Examplefromclimate-basedbudgeting185 % increase in renewable and energy-efficiency, increases in transport and agriculturearesmall

18. Summary of effectiveness • Renewableenergy • Achieving 2020 goalsseemschallenging and dependent on forestindustry as well as the effectiveness of the feed-in-tariffsystem • Agriculture’s N2O emissions • In presentsituationplanned emission reductionappearunrealistic and dependmuch on EU agriculturalpolicy • Transport CO2 emissions • Sofargrowth in transport hasexceeded the benefitsfrombiofuels and technologicaldevelopment. • Communitystructure • Preconditions for effectivenessaremissing outside the MoE:sadministrativesector •  Requiresfurthermeasures in manysectors and in energyefficiency

19. Recommendations of the audit • Furtherincreasingcoordinationbetweenadministrativesectors • Improvingparticipation and transparency • Developing the evaluation of the cost-efficiency of measures and the evaluation of the impacts of the measures on governmenteconomy • Drafting a climatebudget • Identifyingsubsidyelementsthathavecontradictoryeffects • Improvingimplementation in the weakestareas • Clarifying the responsibilities of strategymonitoring and development of climatepolicyevaluation and reporting

20. Conclusions • It is relativelystraightforward to evaluate the progress of policyprocesses and measuresaftertargetsetting, the methodologicalchallenges lie evaluatingbothcost-efficiency and effectiveness • Evaluation of climatepolicy (evenwithinsectoralemissions) complicatedby • Multiplepolicydomains, organisations, levels of governance, scope of emissionsanalysis (national reported vs. global) and lack of scientificprecisionregardingimpacts • The effectiveness of climatepoliciescouldbetentativelyevaluatedbased on theirpotential for change • Incremental vs. system-levelchange • Intervention logic and whetherit is likely to work • Actorperceptions Paula Kivimaa, SYKE

21. Division of climate-basedhumanresources in ministeries Paula Kivimaa, SYKE