Evaluation Working Group Achievements and Future Steps

1. 7th Meeting of the SteeringCommittee of INTOSAI Committee on Knowledge Sharing and Knowledge Services Washington D.C. 15-16 October 2015 Cour des Comptes - Evaluation Working Group - 15/16 october 2015

2. OUR AMBITION • Long term : «help SAIs which practice evaluation or wish to do so by providing them with methodological tools and practical recommendations » (creation of EWG in 1992) • Immediate : approval by KSC Steering Committe of the project proposal and last draft of the guidelines Cour des Comptes - Evaluation Working Group - 15/16 october 2015

3. I – Wherewe are Achievements Major issues II – Key points definition, field, frontiers Standard or guidelines methodology III – Nextsteps Cour des Comptes - Evaluation Working Group - 15/16 october 2015

4. I. Wherewe are Cour des Comptes - Evaluation Working Group - 15/16 october 2015

5. 2010 – XX INCOSAI Achievements (1/4) • Johannesburg, endorsement of the introductory report “Program evaluation for SAIs- A primer” • A website dedicated to the WG activity opened since November 2010 • New question in 2012 : is-it possible, on the basis of the survey, to go further than the 2010 “Primer” ? • Unanimous decision of the group : YES • evaluation constitutes a special category of audit : • Which has links with performance audits • But which includes some specific characteristics : why did a program fail ? How could it be improved ? • And also with a distinctive methodology : association of stakeholders, process of results-sharing Cour des Comptes - Evaluation Working Group - 15/16 october 2015

6. Sept 2012– 4th KSC Achievements (2/4) Decision of the 4th KSC in Sept. 2012 • To aim at a definition of whatisevaluation and whatcouldbecommonprinciples of Program evaluation, based on differentexperiences • To build and to strengthen links with the Performance Audit WG : • Becausethereis no evaluationwithout a solidground of performance audit • Because the process of elaborating standards of audit performance is far more ahead • To draft a practical guide whichcouldbeused, afterapproval, as a guideline for the SAI’swilling to improvetheircapabilities in thisfield Cour des Comptes - Evaluation Working Group - 15/16 october 2015

7. June 2013 – EWG Achievements (3/4) • Elaboration of a preparatory document • Discussion of its content • Agreement on some key points : • It is neither desirable not possible to achieve an ISSAI, at least in a short-term • It is necessary to go further into the definition and the methodology of evaluation : • Some aspects of evaluation clearly don’t belong to performance audit, for example the impact of a program : how can we define and determine what is an impact ? • The process of evaluation involves a wide range of stakeholders : how shall we manage this complexity ? Cour des Comptes - Evaluation Working Group - 15/16 october 2015

8. 2014/2015 – meetings in Paris Achievements (4/4) • Two meetings in Paris respectively in November 2014 and June 2015 • Objective of the meetings : reach a consensus on the distinctive characteristics and methodology of evaluation; modify the preparatory document and summarize it (Nov 2014) • Ultimate Goal : establish guidelines (no « standard ») to be endorsed by KSC, GB 2015 and INCOSAI 2016 (June 2015) • Several amendments both in form and substance were adopted resulting in a new version of the exposure draft and project proposal, approved by EWG members. • The document, primarily intended to be a methodological guide of the different types of evaluations conducted by member countries should focus mainly on the evaluation of public policies. Cour des Comptes - Evaluation Working Group - 15/16 october 2015

9. Crucial Importance of knowledge sharing Major issues (1/3) • A questionnaire to collect case studies and practical information on Evaluation methodology and experiences (elaborated by a sub-group composed of 5 SAIs) was initially approved by the Group in Paris and sent to 191 SAIs. The main lessons learnt (44 SAIs have answered) were presented to KSC in 2012. One SAI out of two deals somehow with evaluation Cour des Comptes - Evaluation Working Group - 15/16 october 2015

10. Crucial Importance of knowledge sharing Major issues (2/3) • A field where definitions are complex, practices are different, comparisons are specifically useful. • Some pitfalls to avoid : focus only on definition problems (even if they are important) ; set wrong ambitions ; lose ourselves in never-ending discussions between standard and guidelines ; complexify instead of clarify … Cour des Comptes - Evaluation Working Group - 15/16 october 2015

11. Crucial Importance of knowledge sharing Major issues (3/3) • Guidelines are even more necessary since evaluation policies could involve specific risks: • close to politics, even to politicalphilosophy • subject to competitionwithsomeunusualactors • far fromSAIsusualmethods • has to deal with new and not alwaysrobust techniques Cour des Comptes - Evaluation Working Group - 15/16 october 2015

12. II. Key Points Cour des Comptes - Evaluation Working Group - 15/16 october 2015

13. Preparatory document June 2013 Key Points (1/3) • Attempt to distinguish evaluation from performance • Evaluation questions if the adequate means have been set up by the public authorities in order to reach its objectives • Evaluation considers both direct and indirect effects • Evaluation aims at querying the objectives of a program at the end of the process • The draft also addresses the methodology of evaluation • Organisation of the evaluation, in relation with the stakeholders • Questions to ask, data to provide, • Planning of the process Cour des Comptes - Evaluation Working Group - 15/16 october 2015

14. Preparatory document June 2013 Key Points (2/3) • Similarity between performance audit/ evaluation: • To go beyond the compliance audit, to measure the links between public measures and policies and their effects on the ground • To assess the “added value” of public policies • Some differences to investigate: • Goal of the evaluation is different : evaluation questions if the adequate instruments were set up to reach some objectives and “in fine” if the objectives of the public policies were worthwhile… • Whereas performance audit focuses on the efficiency and effectiveness between public decisions and actions and their intended effects • A potential “cross fertilization” between those two approaches Cour des Comptes - Evaluation Working Group - 15/16 october 2015

15. Preparatory document June 2013 Key Points (3/3) • Choice of the object of evaluation : • the scope of the evaluation is (or may be) broader than the one of performance audit because its goal is to assess a wide range of actions contributing to a public objective • Methodology : • The evaluation shall address direct and also indirect effects of public decisions ; it shall also wonder if there are some undesirable effects and to what extent a program or a policy can affect other public measures • The evaluation shall “ab initio” and “ex post” take into account the point of view of the stakeholders of a policy and therefore define a process which associates them and allows them to give their feed back. Cour des Comptes - Evaluation Working Group - 15/16 october 2015

16. III. NEXT STEPS Cour des Comptes - Evaluation Working Group - 15/16 october 2015

17. AND NOW THE NEXT STEPS –Four stages in developingINTOSAI GOV Cour des Comptes - Evaluation Working Group - 15/16 october 2015

18. STAGE 1 –PROJECT PROPOSAL • Drafting of the projectproposalby EWG • 15 June 2015: EWG meeting in Paris (discussions on the last drafts of the project proposal and the guidelines on the evaluation of public policies). • 16 June to 22 June 2015 : drafting of the project proposal and the guidelines according to the received comments during the last EWG meeting. • 22 June 2015: new drafts of the project proposal and the guidelines to be sent to EWG members for comment • 6 July to 14 July 2015  drafting of the final version of the project proposal and the guidelines according to the received comments (done by email exchange) • Approval of the Project Proposal by KSC • 15 July 2015: project proposal and guidelines to be sent to KSC for approval. • 15 October 2015: KSC Steering Committee (approval of the project proposal and last draft of the guidelines). Cour des Comptes - Evaluation Working Group - 15/16 october 2015

19. STAGE 2 :exposuredraft • Drafting of the exposure draft by EWG • 16 October to 31 October 2015 : drafting of the exposure draft by EWG • Approval of the exposure draft by KSC • 1 November 2015 : exposure draft to be sent to KSC for approval • 30 November 2015 : approval of the exposure draft by KSC (done by email exchange) • Exposure period of the exposure draft on www.issai.org by KSC • 1December 2015 to 29 February 2016: comments of the exposure draft on the website • Engagements of EWG with limited number of International Institutions and external auditors for comments on the exposure drafts • 1 to 31 December 2015 Cour des Comptes - Evaluation Working Group - 15/16 october 2015

20. STAGE 3 : ENDORSEMENT VERSION (1/2) • Overview of commentsreceived by EWG • 1 March to 31 May 2016 : analysis of the commentsreceivedduring the exposureperiod • 1June 2016 : new draft of the guidelines to be sent to EWG members for comment • 20 June 2016 : EWG meeting (discussions on the new draft of the guidelines) • Drafting of the endorsement version of the guidelines by EWG • 21 June to 31 June 2016 : drafting of the endorsement version according to comments received during the last EWG meeting • 1July 2016 : Endorsement version to be sent to EWG members for comment. • 31 July 2016 : drafting of the endorsement version according to comments received from EWG members (done by email exchange) • 1August to 14 October 2016: Translation of the final endorsement version in the 5 official languages of INTOSAI. Cour des Comptes - Evaluation Working Group - 15/16 october 2015

21. Stage 3 : endorsementversion (2/2) • Approval of the endorsement version by KSC and by the Governing Board • 15 October 2016: Endorsement version to be sent to KSC for approval • November 2016 : Endorsement version tro be sent to Governing Board for approval Stage 4 : endorsement of final intosaigov • XXI INCOSAI, Abu Dhabi, 2016 Cour des Comptes - Evaluation Working Group - 15/16 october 2015

22. Thanks for your attention.INTERNATIONAL@ccomptes.fr, DOMINIQUE PANNIER, SENIOR AUDITOR, DIRECTOR, INTERNATIONAL ReLATIONSJean-FRANçois MONTEILS, Senior Auditor, Liaison officer for EWG Cour des Comptes - Evaluation Working Group - 15/16 october 2015