Program evaluation working group Information report

1. Program evaluation working groupInformation report 5th Knowledge and sharing Committee (KSC) Annual Meeting New Delhi 18-19 september 2013

2. ISC Evaluation : the wayforward Where we come from Where we are Where we could get to PROGRAM EVALUATION WG

3. Wherewe come from • Evaluation working group set in 1992 (XIVth INCOSAI Washington) • The Evaluation Working Group, is composed of 19 SAIs and chaired by France • 1995 XVth INCOSAI Cairo : demand for methodological guide • 1998 XVIth INCOSAI Seoul : draft report on methods and practices PROGRAM EVALUATION WG

4. 2004 XVIIIth INCOSAI Budapest : introductory report (fundamental concepts and examples) 2008 XIXth INCOSAI Mexico : Evaluation related to Knowledge sharing more than Standards 2010 XXth INCOSAI Johannesburg : endorsement of the introductory report “Program evaluation for SAIs- A primer”, proposed by EWG under the auspices of KSC PROGRAM EVALUATION WG

5. Wherewe come from A website dedicated to the WG activity has been open since November 2010. A questionnaire to collect case studies and practical information on Evaluation methodology and experiences (elaborated by a sub-group composed of 5 SAIs) was approved by the Group in Paris in May 2011 and sent to 191 SAIs. Its results (44 SAIs have answered) were presented to last KSC in 2012. PROGRAM EVALUATION WG

6. A large fraction of SAI are practicing different aspects of evaluation PROGRAM EVALUATION WG

7. There is a growing use of participative process PROGRAM EVALUATION WG

8. But still, a minor regular use of external expertise PROGRAM EVALUATION WG

9. Survey general conclusions • Practices seem diverse, show clear signs of converging but also persistent differences due to administrative and political culture • However, the answers confirm that, as long as the mandate of the SAIs is not limited to auditing compliance but extends to analysing the efficiency and effectiveness of actions taken by government administrations, the SAI is capable of assuming a broader scope of analysis, should they be ready to make the necessary changes in their method of work and culture. PROGRAM EVALUATION WG

10. Whatwe have donesince 2012 Paris June 2012 : the WG must go on and prepare methodological guideline for evaluation. March 2013 : a draft summary of the purpose and the content of the methodological guidelines was elaborated by France and sent to participants. Written comments were sent by Switzerland, USA, Gabon and Poland. PROGRAM EVALUATION WG

11. Whatwe have donesince 2012 • Draft document setting main items for the methodological guide for evaluation • Definition(s) • Objectives • Actors and institutional environment for SAIs • Methodology • Dissemination and use of evaluation results PROGRAM EVALUATION WG

12. Whatwe have donesince 2012 • Draft document setting main items for the methodological guide (follow) • Methodology • Choice of subject to be evaluated • Evaluation Planning • Setting the researchable questions • Organizing the evaluation • Process building with stake holders • Scientific tools • Ressources identification • Reviewing results • Counter argument phase • Finalising results PROGRAM EVALUATION WG

13. Last PAS meeting Ottawa May 2013 • According to resolution taken in 2012, the EWG French representative attended the 6th PAS meeting in Ottawa, May 28-29 2013 • From now on, this representative is also the French member of PAS • The purpose was : • To give information to PAS about work progress of EWG • To discuss common points and differences between Performance audit and Evaluation • To clarify the position of Evaluation methodological guidance regarding professionnal standards (ISSAIs) • It was decided to go on cooperation between EWG and PAS and to exchange information and documents. PROGRAM EVALUATION WG

14. Last meeting Paris June 2013 • 10 of the EWG members participated to the Paris meeting on June 25, 2013 chaired by France • Belgium, Finland, France, Gabon, Republic of Korea, Morocco, Poland, Switzerland, • the United States via Webex • Brazil was invited as a guest SAI as the Chair of the INTOSAI Professional Standards Committee • Chile, Costa Rica, El Salvador, Finland, Georgia, Germany, Libyan Arab Jamahiriya, Lithuania, Mexico, Pakistan, Papua New Guinea did not participate but were asked to validate the resolution PROGRAM EVALUATION WG

15. Last meeting Paris June 2013 • The EWG discussed the links between PAS and Evaluation Guidelines • The EWG discussed and amended the draft document proposed in April • The EWG concluded that : • the need for guidelines was still there • the content proposed by the draft document was clear • the elaboration of the document itself would be shared within sub-groups for specific issues • there was no clear reason at this stage to erect an ISSAI for evaluation PROGRAM EVALUATION WG

16. Wherewecouldget to First proposal for KSC september 2013 and INCOSAI Beijing october 2013 Elaborating the Pratical guide within subgroups in 2013-2014 Giving this guide the status of INTOSAI Guideline (cf Communicating and promoting the Value and Benefits of SAIs, presented to KSC by WGVBS) Endorsment by Governing Board 2015 Presentation to INCOSAI 2016 PROGRAM EVALUATION WG