Estimating Project Success

1. Estimating Project Success Estimating Project Success June Verner and Barbara Kitchenham Empirical Software Engineering NICTA

2. Outline • Background • Data • Factors • Data sets • Methodology • Factor analysis • Logistic regression • Results summary • Conclusions and further work

3. Background “Billions of dollars are wasted each year on failed software projects... ….we have a dismal history of projects that have gone awry” [Charettte IEEE Spectrum Sept 2005] • Have been developing software since the 1960s but still have not learned enough to ensure project success • Most project failures are predictable & avoidable • How can we identify these projects early enough to take action?

4. Failures • Most organizations try to hide their failures • Not only monetary loss, but also lost opportunity • A recent “Hall of Shame” includes (in $US millions) • FBI 100 • UK Inland Revenue 33 • Ford Motor Company 400 • Sainsburys 527 • Sydney Water Corp 32.2+ • Other recent Australian problems include: • National Australia Bank AUD200 million write down on failed ERP project, • RMIT’s Academic Management System • Victorian State’s Infrastructure Management System • Continued controversy over the Federal Government’s new sea cargo import reporting system

5. Data - Factors • Literature • Discussions with 90+ developers • Categories • Sponsor • Customer and users • Requirements • Estimation and scheduling • The project manager • Project management • Development process • Developers

6. Data sets • Mostly in-house software developments • North American financial Institution • 42 projects • 45% success rate • Other NE US projects • 79 projects • 71% success rate • Sydney • 42 projects • 78% success rate • Chile 200+ • In house • Developments for third parties

7. Methodology • Correlate all factors with project success • Consider only those at the 95% level • Overall • By groups • Remove factors with large number of missing values • Use factor analysis on reduced set of variables to develop a new set of variables suitable for predicting project success • Use these variables to develop prediction equations on entire data set and by groups • How do these equations compare? • Take original reduced set of correlated factors and develop prediction equations overall and for each of the groups • Compare the results with the equations developed with reduced set of factors.

8. Correlations with project success • Only variables correlated at the 95% level across 3 groups and overall • Sponsor - nil • Customer and Users • level of confidence of customers in the project manager, team members • customers had realistic expectations • Requirements • were requirements completed adequately at some stage • good requirements overall • Estimation and Scheduling • how good were the estimates? • staff added late to meet an aggressive schedule? • Project manager • the PM communicated well with the staff? • how good was project manager? • how well did project manager relate to software development staff?

9. Correlations with project success • Development process • Adequate time was allowed for each of the phases • Development team • How well did the team members work together? • How high was the motivation of the team members? • What was the working environment like?

10. Not included • Sponsor • Project manager given full authority to manage project • Sponsor commitment (2) • Customer and Users • Level of customer involvement (1) • Customer turnover (1) • Large numbers of customers and users • Requirements • Adequate time made available for requirements gathering (2) • Central repository (2) • Size impacted requirements gathering • Scope was well defined (1) • Estimation and Scheduling • Estimate of delivery date used adequate requirements information (2) • Developers were involved in the estimates • Adequate staff assigned to project (1) • Developers were involved in the estimates (1)

11. Not included • Project manager • Project manager background • Years of experience • Experience in the application area • Project manager had a vision of what the project was to do for the organization (2) • Project management • Did the PM control the project? (1) • Staff were appreciated for working long hours (2) • Staff were rewarded for working long hours (2) • Development process • Defined development methodology used (1) • Risks incorporated into project plan (2) • Requirements managed effectively (2) • Developers • Total number of staff (1) • Team members consulted about staff selection (2)

12. Factor analysis • 75% of variance explained with 3 factors • Factor 1- Project manager • The PM communicated well with staff • How good was project manager? • How well did project manager relate to software development staff? • Factor 2 - Customers and requirements • Level of confidence of customers in the project manager & team members • Customers had realistic expectations • Good requirements overall • How good were the estimates? • Factor 3 • Staff added late to meet an aggressive schedule

13. Logistic regression - overall All three factors

14. Logistic regression-by group Marginally better than overall24 wrong versus 23

15. Logistic regression-Original 12 variables-overall

16. Logistic regression-original 12 variables-groups

17. Factors not significant for any of the groups • Project manager given full authority • The project began with a committed champion • The committment lasted right through the project • The sponsor was involved with decisions • Other stakeholders comitted and involved • Senior management impacted the project • Involved customers/users stayed throughout the project • Customers/users involved in schedule estimates • Problems were caused by large by the number of customers/users involved • Reqts were gathered using a particular method • Was the manager involved in estimate? • The project had a schedule • There was a project manager • Years of experience of the PM • Was project manager experienced in application area? • Was project manager able to pitch and help if needed? • The consultants reported to the project manager • Did all the key people stay throughout the project? • Rewards at end of project motivated team

18. Results - summary • Did better with the extracted factors overall • 24 wrong versus 26 • Did better with the extracted factors with groups • 21 wrong versus 24 • Nearly 3 times as many failed project predicted incorrectly • Data set with too few failures problemmatic

19. Conclusions and further work • Would it help if we had values for: • Adequate time was allowed for each of the phases • How well did the team members work together? • How high was the motivation of the team members? • Working environment • Next steps • How best to deal with the missing values? • Use Bayesian networks • What are the missing failure factors? • Don’t believe that in all cases failure is the converse of success