Value For Money (VFM) Audit of IT Systems: Maximizing Efficiency & Effectiveness

1. INTOSAI IT audit training Value For Money

2. VFM audit of IT • Objective • By the end of the module you will be able to: • build a business model; • identify significant IT systems • identify and evaluate evidence of failing IT systems • recognise explanations for IT system failure • assess the development environment to see that sound management practices minimise risks

3. The course schedule • Scope of Value For Money audit • VFM audit method • VFM audit experience

4. VFM audit concepts Value For Money = Performance = 3 Es • Economy is concerned with spending less by minimising the cost of resources of a given quality. • Efficiency is concerned with spending well by minimising resources used for a given output. • Effectiveness is concerned with spending wisely by ensuring that the outputs achieved match the policy or operational objectives.

5. Economy • Failing to take bulk discounts • Using top quality paper for temporary outputs • Using expensive workstations where a dumb terminal would do • Making routine international calls at peak rate • Buying in resources when there is spare in-house capacity • Building your own system when a suitable off-the-shelf system is already available and cheaper, or visa versa. • Delivering IT services in-house when a cheaper outsourcing option is available.

6. Efficiency • Controls that serve no real purpose • Making frequent calls when one would do • Duplication of systems that perform the same function • Entry of information already held by the system

7. Effectiveness • Systems that serve no business function • Unreliable systems • Systems that produce reports that are never used

8. VFM audit objectives • Improving clients’ business management • Validating central guidance • Explain waste • Recommendations for corrective action

9. VFM audit method • Survey • Is it important? • Evidence of success or failure • Is it going wrong? • Explanation • Why is it going wrong? • Recommendation • What can be done now?

10. Survey • Objectives • Organisation • Resources • Performance • Prioritisation • IT Strategy and Environment • Systems

11. Objectives What the client aims to do • Owner • Hierarchical • Link to performance indicators • Link to activities

12. Weighting • 1. Mission=100 • 1.1 40 • 1.1.1 30 • 1.1.2 10 • 1.2 60 • 1.2.1 15 • 1.2.2 45

13. Organisation How the client aims to meet objectives • Policies and standards • Control environment • Culture • Activities

14. Activities • Produce outputs • Business units • Cost centres • Link to objectives • Link to performance indicators • Supported by information systems

15. Information systems • Support activities • Contribute to objectives • Monitor progress • Record resource usage • IT Strategy and Environment • IT Governance Maturity Model

16. Resources • Inputs to activities • Resource management • Internal versus external

17. Performance • Impact versus output • Performance regime • Performance indicators • Benchmarking • Stakeholders

18. Performance regime • Key indicators • Clear responsibilities • Good resource management system • Regular review • Decisive action • Targets

19. Performance indicators • Link to objectives and activities • Measurable • Comprehensive • Consistent • Comparable • Verifiable

20. Benchmarking • Activities • Impacts • Across time • Across organisations

21. Stakeholders • Customers • Politicians • Journalists • Academic and professional bodies

22. D A L E T K R O S S H E Framework for Audit and Control of Effectiveness Objectives Inputs ACTIVITIES Feedback Review AIM TARGET C O S T S Outputs Impacts Performance Indicators

23. Study selection • Poor performance • High cost • Strategic importance • Management weakness • Systematic failure • Relevance to many clients

24. Materiality of IT Programme costs IT costs AdministrationExpenditure

25. Importance of systems Programme delivery SYSTEMS IT Administration

26. Place of IT in studies • Part of explanation • Large projects • Comparative studies • Gross or systematic failures • Central guidance • Emerging technology

27. VFM audit method - Evidence • Survey • Is it important? • Evidence of success or failure • Is it going wrong? • Explanation • Why is it going wrong? • Recommendation • What can be done now?

28. Quality user dissatisfaction unreliable poor integration costly to run or maintain disputes with suppliers Time abandoned or delayed systems Cost expensive systems Evidence

29. User dissatisfaction - sources of evidence • Post implementation reviews • System owner • Survey • Interviews • Help desk • System usage statistics

30. User dissatisfaction - explanation • Strategy formulation and review • Requirements capture • Inadequate quality control • Operational management • Training and awareness

31. Unreliable systems - sources of evidence • Operations manager • System owner • Support staff • Maintenance records • Error logs

32. Unreliable systems - explanation • Design standards • Development standards • Maintenance • Operational management • Infrastructure

33. Poor integration - sources of evidence • Duplicate entry • Complex data conversion • Data administrator

34. Poor integration - explanation • IT strategy • End user computing • Procurement control • Development control

35. Cost overruns -sources of evidence • System owner • Business case • Project board minutes • IT steering committee • Project control documents • Management accounts • Post implementation reviews

36. Cost overruns - explanation • Investment appraisal • Project management • Design standards • Development standards • Operations management

37. Delays - sources of evidence • Strategic plans • System owner • Project board minutes • IT implementation schedules • Business case

38. Delays - explanation • Unrealistic timetable • Project management • User opposition

39. Failed projects - sources of evidence • IT strategies • IT steering committee • Post implementation reviews • Potential users • Finance department

40. Failed projects - explanation • Business case • Requirements capture • Project management

41. Costly maintenance - sources of evidence • Comparison with other systems • Resource management system • Change management records • System owner

42. Costly maintenance - explanation • Requirements capture • Flexibility • Development standards • Skills shortage

43. Supplier disputes - sources of evidence • Correspondence with supplier • Interview owner • Records of meetings

44. Supplier disputes - explanation • Inadequate specification • Unrealistic bid • Service level agreements

45. Lack of evidence is evidence of danger • No business plan • No IT strategy • No business case • No owner / users • No project plan or budget • No resource monitoring • No post implementation reviews

46. VFM audit method - Explanation • Survey • Is it important? • Evidence of success or failure • Is it going wrong? • Explanation • Why is it going wrong?

47. Explanation • IT standards • IT strategy • User involvement • Business case • Procurement • Project or operational management • Business continuity • Obsolescence

48. Explanation - IT standards • Strategy • Management • Design and development • Technical integration • Change management • Standards ignored?

49. Explanation - IT strategy • No link to business plan • Ignored • Unrealistic • Uncoordinated

50. Explanation - User involvement • No system owner • Poor requirements capture • Inadequate training and awareness • No formal user acceptance