Supreme Audit Institution Performance Measurement Framework (SAI PMF)

1. Supreme Audit Institution Performance Measurement Framework (SAI PMF) Presented by: Yngvild H. Arnesen, World Bank The Exchange, Abu Dhabi, 13-15 May 2013

2. Presentation Outline • What is the SAI PMF and why is it developed? • Why do a SAI PMF, and how? • Practical example: Audit Service Sierra Leone • What is next on the SAI PMF?

3. 1. What is the SAI PMF and why is it developed?

4. What is the SAI PMF? • A Performance Measurement Framework tailored to SAIs – developed by INTOSAI • Intended to give a holistic, high level assessment of SAI performance against established INTOSAI points of reference: • The International Standards of Supreme Audit Institutions (ISSAI) framework; • The Framework on the Value and Benefits of SAIs; • Other INTOSAI guidance material and international good practices. • Combination of objective measurement and qualitative assessment • Assesses SAI performance in the country context and contributes to identifying the value and benefits of the SAI to citizens

5. Why is the SAI PMF developed? INTOSAI Framework on the Value and Benefits of SAIs adopted 2010 Increased focus on use of country public financial management systems within the international donor community Full set of International Standards of Supreme Audit Institutions (ISSAIs) adopted by INTOSAI in 2010

6. How is the SAI PMF beingdeveloped? Mandated by Johannesburg Accords, INTOSAI Congress, 2010 SAI PMF Task Team • INTOSAI Regional Organizations: African Organization of English Speaking SAIs(AFROSAI-E) and African Organization of Francophone SAIs (CREFIAF) • SAIs: Bahrain, Brazil, Chile, Germany, India, Mexico, Russia, United Kingdom • Development Agencies: Inter American Development Bank • Coordinator:INTOSAI-Donor Secretariat @ IDI SAI PMF Reference Group • INTOSAI Regional Organizations: Pacific Association of SAIs (PASAI) • SAIs: Canada, Denmark, Estonia, European Court of Auditors, Japan, Netherlands, Norway, Peru, Sweden • Development Agencies: African Development Bank, OECD, European Commission, World Bank • Civil Society Organizations: International Budget Partnership, Global Reporting Initiative (GRI), PEFA Secretariat Responsibility of Working Group on the Value and Benefits of SAIs

7. SAI PMF: Evidence Based Assessment of Performance Holistic, judgmental assessment of the SAI and its environment – including factors not covered by the indicators Objective measurement to inform qualitative assessment and track progress over time

8. SAI Performance Measurement Framework SAI Reporting SAI Impact SAI Capability SAI Environment SAI STRATEGY • SAI Reporting C. Strategy for Organizational Development B. Independence and Legal Framework SAI CORE BUSINESS Value and Benefits of Supreme Audit Institutions D. Audit Standards and Methodology F. Human Resources and Leadership G. Communication and Stakeholder Management E. Management & Support Structures SAI SUPPORT SERVICES Country Context, Governance and Public Financial Management

9. 2. Why do a SAI PMF, and how?

10. Why do a SAI PMF assessment? • Manageownperformance: • Wherearewe? • Monitor progress over time • Implementation of International Standards for Supreme Audit Institutions (ISSAIs) • Demonstrateaccountability • Provide evidence to donors on SAI performance

11. Voluntary use Who decides toinitiate a SAI PMF? Head of SAI Who decides when to do a SAI PMF? Head of SAI Head of SAI Who decides how to do a SAI PMF? Head of SAI Who decides if, when and how to publish a SAI PMF?

12. How will the SAI PMF Assessment be conducted? 1. SAI SelfAssessment 2. Peer Review 3. ExternalAssessment SAI Involvement • The approach will not necessarily affect how the assessment is carried out… • … but the SAI may use the results for different purposes depending on the approach

13. Issues in measuring SAI performance • SAI PMF is not designed to judge and rank SAI against each other, but aims to identify strengths and weaknesses of the individual SAI • SAI’s should not be held accountable for matters over which they have no controlSolution: Clear distinction between factors within and outside the SAI’s control • Some aspects of SAI performance are difficult to measure, especially ‘quality’Solution: Qualitative analysis in performance report, focus on processes and systems that should result in high quality audits

14. Issues in measuring SAI performance (2) • The mandate of a SAIs differs across countries and may not be aligned with the ISSAIs. Solution: Assess the SAI against its own mandate (but assess its mandate against ISSAIs) • The financial resources allocated to the SAI may not be sufficient to carry out its mandate; not appropriate to punish SAI for low performance on ‘coverage’, professional training etc.Solution: Assess SAI against its legal mandate, assess its reporting to parliament on coverage, include pertinent issues in performance report

15. 3. Practical Example: Audit Service Sierra Leone

16. Piloting the SAI PMF • First round of pilots of SAI PMF: • Second half of 2012 • Three SAIs: Djibouti, Norway, Sierra Leone • Carried out by SAI PMF Task Team • Pilot of Audit Service Sierra Leone: • Field work in October 2012 in cooperation with AFROSAI-E • Document analysis, review of audit files, interviews • Entire process: about six months

17. Independence & Legal Framework SAI Reporting Results – Sierra Leone • Strategy for Organizational Development • Strategy guides organizational development & delivery of mandate • Monitoring weaknesses limits continual improvement of audit methods & public reporting of SAI performance • Financial audit coverage • Timely submission & publication of audit reports • Selection of performance audit topics • Performance audit coverage • Follow-up of audits • Reporting performance • Constitutional independence of SAI & AG • Mandate for public sector • Financial, compliance & performance audit • Freedom to publish reports • Financial independence: • Budget process & receipt of funds • Audit Standards & Methodology • Risk-based financial & compliance audit manual • Training based on manual • Quality assurance system • Performance audit manual • Documentation & quality control • Supervision & review • Performance audit capacity • Planning & monitoring delivery & cost of audits Management & Support Structures Human Resources & Leadership Communication & Stakeholder Management • External training support • Link staff plan to mandate • Staff accountability & reward for performance • Attract & retain staff • Managerial vacancies • Training resources • Code of ethics, job rotation, structures • Monitoring staff utilization • Infrastructure & IT • Reporting to PAC • Working with PAC • Radio debates • Internal comms. • Comms. with audited entities

18. Internal Factors Impacting on SAI Performance Managerial vacancies Planning & monitoring use of (staff) resources Staff motivation & reward Staff accountability for performance Internal communications

19. External Factors Impacting on SAI Peformance – Sierra Leone

20. Value & Benefits of the Audit Service to Citizens Promoting fiscal transparency and accountability Deterring, preventing & detecting corruption Communicating with & engaging citizens: demonstrating relevance to stakeholders

21. Added Value for Audit Service Sierra Leone • Identification of strengths and weaknesses– knows what to focus on in future • Knowledge about weaknesses caused by internal factors which are under the SAI’s control… • … and weaknesses which are caused by external factors that the SAI may try to influence • Foundation for communicating value and benefits of the Audit Service • Used as baseline for tracking performance over time – linked to external funding • Used to inform design of capacity building interventions

22. 4. What is next in the development of SAI PMF?

23. SAI PMF Roll-out • INTOSAI Congress 2013: Official exposure draft • Training: For assessors and future trainers – in all INTOSAI regions during 2013/2014 • Piloting:Around 20 volunteer SAIs with different characteristics • Guidance material: Prepared by IDI • Support function @ IDI

24. For more information: www.idi.no E-mail: INTOSAI.Donor.Secretariat@idi.no

25. Thank you!