6th INTOSAI KSC Steering Committee Meeting in Cairo

1. 6th INTOSAI KSC SteeringCommittee Meeting in Cairo 15 October 2013

2. Evaluation WG : an old ambition, a long history • Creation in 1992 with 1 goal : «help SAIs which practice evaluation or wish to do so by providing them with methodological tools and practical recommendations » • Achievements : • 2010 - XX Incosai in Johannesburg, endorsement of the introductory report “Program evaluation for SAIs- A primer” • A website dedicated to the WG activity opened since November 2010

3. Survey : one SAI in two deals somehowwithevaluation • A questionnaire to collect case studies and practical information on Evaluation methodology and experiences (elaborated by a sub-group composed of 5 SAIs) was approved by the Group in Paris in May 2011 and sent to 191 SAIs. Its results (44 SAIs have answered) were presented to KSC in 2012.

4. New start in 2012 • New question in 2012 : is it possible, on the basis of the survey, to go further than the 2010 “Primer” ? • Unanimous decision of the group : YES • Why ? Because it appeared that evaluation constitutes a special category of audit : • Which has links with performance audits • But which includes some specific characteristics : why did a program fail ? How could it be improved ? • And also with a distinctive methodology : association of stakeholders, process of result-sharing

5. Decision of the 4th KSC in Sept. 2012 • To aim at a definition of what is evaluation and what could be common principles of Program evaluation, based on different experiences • To build and to strengthen links with the Performance Audit WG : • Because there is no evaluation without a solid ground of performance audit • Because the process of elaborating standards of audit performance is far more ahead • To draft a practical guide which could be used, after approval, as a guideline for the SAI’s willing to improve their capabilities in this field

6. What has been donesincethen ? • Elaboration of a preparatory document • Discussion of its content at the meeting of the EWG in June 2013 • Agreement on some key points : • It is neither desirable not possible to achieve an ISSAI, at least in a short-term • It is necessary to go further with the definition and the methodology of evaluation : • Some aspects of evaluation clearly don’t belong to performance audit, for example the impact of a program : how can we define and determine what is an impact ? • The process of evaluation involves a wide range of stakeholders : but how shall we manage this complexity ?

7. Issues addressed in thispreparatory document • Attempt to distinguish evaluation from performance • Evaluation wonders if the adequate means have been set up by the public authorities in order to reach its objectives • Evaluation considers both direct and indirect effects • Evaluation aims at querying the objectives of a program at the end of the process • But the draft also addresses the methodology of evaluation • Organisation of the evaluation, in relation with the stakeholders • Questions to ask, data to provide, • Planning of the process

8. A new document preparedduring S1 2014 Whose purpose is to take into account the first remarks of the EWG’ members Which was sent at the beginning of Summer Asking for comments in September / October before the meeting of the EWG to be held in Paris the 17th of November

9. Key points of the preparatory document(1/2) : objective of evaluation • Similarity between performance audit/ evaluation: • To go beyond the compliance audit, to measure the links between public measures and policies and their effects on the environment • To assess the “added value” of public policies • Some differences to deepen : • Goal of the evaluation is different : evaluation wonders if the adequate instruments were set up to reach some objectives and “in fine” if the objectives of the public decisions were worthwhile… • whereas performance audit focuses on the efficiency and effectiveness between public decisions and actions and their intended effects • An unavoidable “cross fertilization” between those two approaches

10. Key points of the preparatory document(2/2): object and methodology • Choice of the object of evaluation : • the scope of the evaluation is (or may be) broader than the one of performance audit because its goal is to assess a wide range of actions contributing to a public objective • Methodology : • The evaluation shall address direct and also indirect effects of public decisions ; it shall also wonder if there are some undesirable effects and to what extent a program or a policy can affect other public measures • The evaluation shall “ab initio” and “ex post” take into account the point of view of the stakeholders of a policy and therefore define a process which associates them and allows them to intervene

11. Questions, comments and suggestions alreadyreceived (1/3) • Q : What is the link between the guidebook of 2010 and this draft ? • A : the guidebook of 2010 proposed a general definition of evaluation, this draft aims to go into detail and to be more specific about the links between performance audit and evaluation (whether evaluation is a distinct form of audit or a branch of performance audit) • Q : What are the main issues to be addressed in a methodological guide ? • A : of course we do not have to sum up all the aspects of the audit’s process and we should mainly clarify the points which are specific of an evaluation, for instance the association of the stakeholders

12. Questions, comments and suggestions alreadyreceived (2/3) • Q : is there a need for a guide and shouldn’ t we take advantage of the existing literature ? • A : of course it is useful to develop exchanges on this topic between the SAI’s and it is also mandatory to base a guide on existing practices. But the goal of the EWG does not consist only in exchanging best practices but also in establishing common principles and in delivering a framework for evaluations. • A : of course we need a « bottom-up » process • Q : why shouldn’ t we focus on developping abilities of the SAI’s to carry out evaluations ? • Of course this has to be made, but this development should be based on a common definition.

13. Questions, comments and suggestions alreadyreceived (3/3) • Q : is evaluation a specific form of audit or a branch of performance audit ? • A : ISSAI 3000 clearly states that « performance auditing examines the economy, the efficiency and the effectiveness of government programs and organizations ». And it also defines effectiveness as the achievement of the policy objectives • But it also says that a performance audit will not necessarily seek to reach conclusions on all three aspects ; and that the objectives should be the starting point for performance auditing, • Instead, the relationship between ouputs and outcomes, on the one hand, outcomes and policy objectives, on the other hand, is the core of program evaluation, as well as the question of the relevancy and the consistency of the objectives • No matter how program evaluation is specific (completely or partially), the subject is to deepen its distinctive features

14. And now : the nextsteps • Meeting in Paris in November 2013 • Objective of the meeting : reach a consensus on the distinctive characteristics and methodology of evaluation; modify the preparatory document and summarize it • Ultimate Goal : Come to methodological guidance (no « standard » at the moment) to be endorsed by KSC, GB 2015 and INCOSAI • We do not have to reach an agreement on every item of the draft. • But it would be incomprehensible, after the introductory report of 2010 and the « boosting decision» of 2012 to let the discussions of the EWG unfinished

15. Thanks for your attention.For further information, do not hesitate to contact us jralventosa@ccomptes.fr, Director International Departmentekesler@ccomptes.fr, Senior Auditor, Liaison officer for EWG