Using Periodic Audits to Prevent Catastrophic Project Failure

1. Using Periodic Audits to Prevent Catastrophic Project Failure Dr. Paul Dorsey Dulcian, Inc. May 22, 2008

2. The Vasa • Early 17th century (1628) • “The greatest ship of all time” • 3 years to build • 2 gun decks with 64 cannons • King Gustavus Adolphus of Sweden set the measurements. • 1000 trees were required • Triple laminated oak walls (18in/46cm thick) • Mast = 190 ft/57m • Length = 201 ft/61m • Cost = 5% of Sweden’s GNP

3. Maiden Voyage • Set sail on a beautiful summer day • August 10, 1628 • Passed the royal palace of Stockholm • Fired proud salutes from cannons • Sailed 1400 yards • Gust of wind came • Ship sank in less than 1 minute

4. Why is the Vasa relevant? • What sank the Vasa sinks FMIS projects. • We are still building Vasas. • We can’t stop bad decisions. • We CAN stop ignoring them. • If you spend $1M and fail, it is bad. • If you spend $100M and fail, it impacts the whole organization • If you spend $1B and fail, it impacts the country. • If you are going to fail - fail cheap.

6. The Essence of Project Management • Leading kids on a hike • From time to time, climb a tree • Check for obstacles • Adjust direction • Call everyone together • "LET'S GO!"

7. Periodic Audits • Critical success factor for failure prevention • Must be external • Developers cannot self-assess. • Big, substantial effort • Weak audit is worse than useless • Provides illusion of safety

8. Audit Costs • Delay project • Expensive • 5%-10% of project cost • Intrusive • Annoying • Politically costly

9. Team response to audit • Project Manager • "Why don't you trust me?" • "Waste of time" • Developers • "We would rather be coding." • Team may feel threatened… • If the team feels threatened, they probably have reason to be. • If the team welcomes the audit…. • Sign of professional maturity

10. Audit Benefits • Early detection of failure • If a $300 million project fails after $20 million, that is a huge savings. • Validation of system architecture • Second set of eyes • Give team time to take stock • Mid-project course correction

11. Auditor must be told that there will be no chance for follow-on work. Otherwise the audit is suspect To enforce independence: No economic attachment to the results No incentive to skew the results No relationship to the development team During the audit: Limit contact with development team "Stockholm Syndrome" After the audit, auditors can help with the project plan. Auditor is the agent of the person who hired him/her (no one else) Only reports to the contract owner are objective. No professional standard or certification for IT auditors exists. Contract creates objectivity. Auditor Independence

12. Audits are never 100% objective • Auditors bring their own biases. • There are IT "religions." • .Net vs. Java • "Thick database" vs. middle tier logic • Service Oriented Architecture (SOA) • Repository-based development • Business rules • Agile • A professional with a different perspective can still detect a "Vasa."

13. Justifying Audit Cost • An extensive audit requires approximately 10% of the total project cost. • Industry project failure rate is ~ 60%. • Example: Audit at the halfway point of a $1 million project • Cost: (50% x 1,000,000) x 10% = $50,000 • Benefit: (50%) x 1,000,00) x 60% = $300,000 • Given the high failure rate, audits are very cheap insurance. • With other benefits, it is obvious that audits are a good deal.

14. Finding the right auditor • Not internal • Not from the same company as the developers • Not dedicated auditors • Must be real developers, DBAs, architects • Must have built systems of similar scope and subject area

15. The Audit Team • Project Manager • Experience in the subject area with projects of similar scope • Database Administrator (DBA) • Experience with same platform and similar database size • Application Architect • Skill in the same area (Java, .Net, Oracle, etc.) • Security Specialist • Military, financial system or health care experience

16. Structure of the Audit • Knowledge transfer • Deep understanding • As if auditor were taking over the project • Understand the system before evaluating • Infrastructure audit • Evaluate the existing infrastructure to support system • Every area needs to be assessed. • Ability to meet current and future user requirements • Auditor must understand the requirements • Financial review

17. A. Knowledge Transfer Allows auditors to understand the entire system architecture as if they were taking over system development. The following areas should be reviewed for the knowledge transfer portion of the audit: System Overview Data Model Walkthrough Review/Identify Transaction Use Cases Review User Interface Code Architecture Review Reporting Requirements/Architecture Review System Architecture Review System Installation/Upgrade Mechanism. Internal Control review User Access review Handling system bugs/enhancements Multi-Lingual capabilities Basic System Requirements Process Flows Custom Framework Performance Standards Training B. Infrastructure Audit Examine from technical soundness perspective. Compare to current industry best practices; document any discrepancies. System and User Documentation Data Model Audit Database Review User Interface (UI) Architecture Review Distributed System/ETL Audit Security Audit Hardware/Software/Networking Review Backup/Recovery Procedures Appropriateness of system upgrade mechanism C. Ability to Meet Current/Future Requirements Examine the current system requirements, identify use cases, and review for suitability, specifically: Compare documented requirements to the existing use cases and how they are handled. Assess user satisfaction with the existing system. Are existing backup/recovery procedures sufficient to meet maximum downtime requirements? Assess system flexibility to meet new requirements. D. Financial review Review how resources have been spent over the lifetime of the project including budgeted vs. actual expenditures Detailed Structure of Audit

18. Knowledge Transfer • "Seek first to understand." Stephen Covey • Learn enough to take over the process: • Architecture • Data Model • Use Cases • Report Audit • Configuration Management • Internal Controls • Documentation • Training • System may be bad enough that this is not possible. • Do it anyway. • Preventing the Vasa from sailing is hard work.

19. Compare what was learned in the knowledge transfer with best practices Each area needs to be assessed: Documentation Data model Database design User interface architecture Security Backup & Recovery Configuration Management Internal Controls Identify weaknesses in each area: Corrective actions Exposure Controls needed One mistake can sink the Vasa. System won’t scale Security hole Inflexible design Infrastructure Audit

20. Ability to Meet Requirements • Requires careful use case documentation • Assess user satisfaction • Sit with users • Survey • Request queue is not a good measure. If system is poor, users give up. • Assess each use case • Functional requirements • Performance • Downtime • Future requirements • Flexibility • Deployment

21. Stewardship Audit If requirements change, price can balloon. Does this make sense? Sunk costs are "somewhat" relevant Measure decision quality Financial History Financial Review

22. COTS Project Audits • Not very different from custom projects • COTS projects fail just as often. • Review COTS architecture • Be careful about how well COTS satisfies requirements: • COTS customizations can be VERY expensive. • Customized COTS cannot be upgraded.

23. Audit Report • 2-5 page Executive Summary Report • Are we OK? • 10-15 page CIO Report • Are we OK? Why? • 100 page detailed report • What we did • What we found • What needs to be fixed • Next steps

24. Acting on the Audit Report • If report concludes "Vasa…." • Get a second opinion • Let the development team respond • Sunk costs are sunk costs. • The amount of money budgeted for the project is irrelevant. • "Upgrade" is a way to change direction without admitting failure.

25. Case Study: Ethiopia FMIS • Harvard University Project • Small part of major financial reform • $38 M USD over 12 years • $3 M USD over 3 years for IT (very frugal) • Harvard was ending the project and wanted to assess quality of system. • Custom development project • Dulcian was brought in to do the audit.

26. Audit Scope • Standard audit • As described in this presentation • Total knowledge transfer and team back-up • Support for any “what if?” scenarios • Total system back-up

27. Audit Recommendations • Maintain current system • Upgrade system internals • Business rules approach • Oracle DBMS • Ultra-light Web architecture

28. Audit Results • Government and Harvard accepted recommendations. • Maintain/evolve current system $1.5M USD • Redesign architecture $1.5M USD • Dual nature of the audit (audit + handoff) made existing team very uncomfortable. • Top three IT people resigned.

29. Conclusions • Audits don't prevent failure; they just catch failures earlier in the process. • Audits don't replace good design. • The audit may only help fail more small projects rather than one big project. • Audits are resource intensive, expensive, annoying, politically charged. • But they are cheaper than not doing them at all in the long run. • Auditors must be kept independent. • No follow-on work • Both COTS and custom projects need audits.

30. Result of Audit • Quite a good design • Excellent documentation • Very good developer coding • Supported current requirements • Some architecture portions needed modification. • Database design issues • No Foreign Keys • Odd design (inherited from previous team) • Poor performance for parts of system • Scalability questions • Would not work on Ethiopian internet due to slow/unreliable connections

31. Contact Information Dr. Paul Dorsey – paul_dorsey@dulcian.com Dulcian website - www.dulcian.com Design Using UML Object Modeling Designer Handbook Developer Advanced Forms & Reports Latest book: Oracle PL/SQL for Dummies