Approach of the Office of the C&AG to Examining Value for Money

1. Approach of the Office of the C&AG to Examining Value for Money Seamus McCarthy Deputy Director of Audit Office of the Comptroller and Auditor General Irish Evaluation Network Friday 14 December 2007

2. Objectives • Place VFM examination work by OCAG in the context of accountability and evaluation structures in Ireland • Outline how evaluation challenges are dealt with in VFM examination work • Briefly survey some possible future developments for OCAG assessment of VFM

3. C&AG’s VFM reporting mandate • C&AG may examine and report on • the economy and efficiency achieved by audited bodies in the acquisition, use and disposal of resources • the adequacy of the systems, procedures and practices used by audited bodies to evaluate the effectiveness of their operations • C&AG may make such comparisons as he considers approporate (including comparisons of systems, procedures and practices)

4. Evolution of the mandate • C&AG role is part of the Westminster model of government • Original focus on financial audit • Appropriation accounts system • State agencies’ accounts • Progressively more reports on cases of loss, waste or extravagance e.g. • Project cost overruns • Fishery harbours • Prison planning • Project audits: 1990-1993 • Comptroller and Auditor General (Amendment) Act, 1993

5. Dáil Éireann Minister for Finance Minute Report Information The accountability cycle Committee of Public Accounts Departments/ agencies Public hearing Information Report Comptroller and Auditor General

6. Public auditing principles • Independence • Objectivity/freedom from bias • Wide scope of mandate • Freedom to report publicly • Authority

7. Certification audit Statutory/obligatory Output is a certificate/pro forma Accounts submitted for audit Numbers based Standards set Set audit procedures Answers are clear Ongoing relationship with auditees’ staff Reporting Discretionary Output is a report Nothing formal to ‘audit’ Narrative/analysis based Standards must be developed No fixed procedures No ‘right’ answers New relationships with auditees’ staff Certification audit vs Reporting

8. Certification audit Financial management reporting Value for money reporting Organisation of OCAG C&AG Director of Audit Director of Audit Secretary and Director of Audit Corporate Services Vote audits Semi State audits VFM exams

9. Strategic reporting issues • Reporting governed by ethical/professional standards • Quality assurance • Expert input • Internal review/audit of working papers • Challenge/clearance process • ‘Cold’ review • Resources • Internal staffing • Developing capacity • Consultants • Ensuring independence/objectivity • Understanding of requirements for VFM reporting

10. Stages in the examination process Selection of examinations Planning Investigation Reporting Follow-up

11. Selection of examinations • C&AG exercises discretion • Strategic themes/priorities e.g. • Public sector management reforms • PPPs • Health sector • Matters arising (from audits, controversies, etc) • Consultation with PAC • Review of potential reporting issues based on • Risks to value for money in the body under review (performance risks) • Potential added value of a C&AG report • Feasibility

12. Some recent reports • Schools PPP • Waste management in hospitals • Rent supplement • The Financial Regulator • Follow up review of examinations carried out 1997-2000 • eGovernment • Drug addiction treatment and rehabilitation

13. Examination planning • Scope setting - bound by statutory mandate • Issues to be examined – may include financial management issues • Time frame • Financial scope • Analytic framework development • Liaison with auditee(s) • Tests of data availability • Time and budget planning • Formal review of plan by C&AG

14. Identify social needs Set objectives Budget Budgeting Financial outturn Financial results Activity management Capacity management Planned inputs Measures of resource use Activity measures Inputs (resources) Planned activities Activities/processes Planned outputs Outputs Planned results Results Planned impacts Impacts Public sector business model Output measures Business targets Socio/economic statistics

15. Analytic framework • Core VFM concepts (economy, efficiency, effectiveness) are very high level • Risk-based prioritisation of key performance areas • For each performance area, identify • Performance standard in terms of 3 Es (+ equity) • Performance measure(s) • Target(s) • Focus on performance of relevant bodies – things they control or influence, rather than context indicators • Framework set at outset, but may evolve in light of information

16. Engagement with auditees • Objectives • Brief management about plans, and get their views • Identify data sources/data holders • Agree working procedures • Resolve difficulties • Inform management about progress • Share findings (no surprises principle) • Get the management perspective

17. Investigation • Statutory access rights • Data sources • Official documents • Literature search • Administrative data e.g. databases • Semi-structured interviews (with managers, staff, clients) • Questionnaires/surveys • Sampling of cases (e.g. for process review) • Data handling procedures

18. Analysis • Data checking/validation incl. system review • Forms of analysis used • Exploratory data analysis • Descriptive statistics • Trend analysis • Regression • DCF-based financial models • Activity-based costing • Review/audit of results • Analysis of professional judgements results

19. Reporting • Target audiences: members of the Dáil; the public; public service managers • Style: non-specialist; concise but comprehensive; clear • Conclude on examination questions • Include • recommendations for improvement • examples of transferrable good practice • Formal clearance of drafts • two (or more) rounds of clearance

20. Follow-up • Publication of report • Briefing/update for C&AG for appearance at PAC meeting • PAC hearing • Follow-up review or re-audit • Potential impacts (during or post examination) • Expenditure savings • Greater efficiency • Improved service • Better systems for evaluating effectiveness

21. Possible future developments • Review of C&AG’s legislative mandate • Reporting of transferable good practice • Ad hoc reference groups for specific evaluations • Development of performance reporting (e.g. Output Statements, VFM reports) • Changing structure of public sector – new agencies, inspectors, regulators, PPPs, community/voluntary sector, etc. • Revised performance models/accountability regimes • New issues e.g. gaps in service provision; boundary disconnection; contract/service level management

22. Discussion