Management of Large Software Projects

1. Management of Large Software Projects Tathagat Varma MindTEK, 25-Mar-2004

2. Disclaimer • Most of what I quote is from standard textbooks & research firms • A little of what I say is from my own limited experience. I am still learning Software Project Management ! • I am not specifically advocating any specific process or methodology • Some principles can also be applied in smaller projects • Apply your judgment as per your own requirements and situation

3. Topics not covered • Project Management • Project Structure, Organization of roles and responsibilities • Long-distance Project Management  • People Management • Performance Management • Managing Cross-cultural teams • Quality Management • Detailed discussion on Software Management Processes and Software Engineering Processes • Using Measurement and Metrics for Project Management • Technology Management • Configuration Management

4. Agenda • Economics, Success Rates and Effort for Large Software Projects • Reduce Software Size • Managing Late Changes • Staffing and Recruitment • Productivity • Misc. Topics • My Experiences • Q&A

5. Reality Check #1

6. Reality check #2

7. Project Outcome • Standish categorizes projects into three resolution types: • Successful: The project is completed on time and on budget, with all features and functions originally specified. • Challenged: The project is completed and operational, but overbudget, late, and with fewer features and functions than initially specified. • Failed: The project is canceled before completion, or never implemented.

8. Are we improving ?

9. Reasons behind this improvement • Standish Chairman Jim Johnson says, “The primary reason is the projects have gotten a lot smaller. Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward.” • “People have become much more savvy in project management,” Johnson says. “When we first started the research, project management was a sort of black art.

10. Reasons… • The Standish Group predicts that most projects started and resolved this year (2001) will be “microprojects”: ones lasting no more than four months and run by four people. CHAOS research shows minimization as a key factor in successful projects. The microproject is the ultimate in minimization. Many last only three to four weeks. Don't confuse microprojects with milestones—microprojects live and die on usable deliverables.

11. So, what is a “successful” software project ? • Booch – “A successful software project is one whose deliverables satisfy and possibly exceed the user's expectations, was developed in a timely and economical fashion, and is resilient to change and adaptation”

12. What makes a project “successful” ? • Recipe for Success: CHAOS Ten • Executive Support 18% • User Involvement 16% • Experienced Project Manager 14% • Clear Business Objectives 12% • Minimized Scope 10% • Standard Software Infrastructure 8% • Firm Basic Requirements 6% • Formal Methodology 6% • Reliable Estimates 5% • Other criteria 5% • The Chaos Report (2000) argues that 97% of successful projects have an experienced project manager at the helm.

13. Project Sizes • Size as team strength could be : • Trivial Size: 1 person • Small Size: 5 people • Moderate Size: 25 people • Large Size: 125 people • Huge Size: 625 people • The more the size, the greater are the costs of management overhead, communication, synchronizations among various projects or modules, etc.

14. Why large projects ? • Software and System complexity has gone up manifold in last 10-15 years when two software engineers could make a ‘software’ over a weekend in their garage – also, there were no customer / marketing pressures to deliver fast • Now, complex systems needs a huge amount of software delivered very fast • If schedule is not a constraint, projects could still be delivered by a relatively smaller team

15. ‘Economies’ of Large Projects • Contrary to most manufacturing processes, the more software you build, the more expensive it is. • For example, for a given application, a 10 KLOC software solution will cost less per line than a 100 KLOC software solution. How much less ? Assume that a 100 KLOC system requires 900 staff-months for development, or about 1.37 hrs / LOC. If the same system were only 10 KLOC and all other parameters were held constant, this project would be estimated at 62 staff-months, or about 0.87 hour / LOC.

16. Economics… • So, it is better to reduce the size of a project as much as possible. • Project Size could be reduced by • Reducing software size • Reducing project team size

17. Success by Project Size

18. Characteristics of Large Projects • Substantial management overhead – need dedicated project manager and sub-project managers • Need lot of planning and effort to synchronize project-level and sub-project level workflows and balance resources among various teams • Needs very good processes to manage the workflow and quality because there are many ‘average’ people in a large team – the probability of recruiting, maintaining and retaining a large number of exceptional people is small !

19. Planning Large Projects • Finalize the Architecture first, and manage it continuously • Plan multiple iterations, if possible • Identify dedicated roles to manage project’s technical and management progress and control • Automate the CM, change and defect tracking environment • Reduce Project Size

20. Managing Software Architecture • Software Architecture becomes extremely important for a large product / project • Ideally, a full-time dedicated Software Architect must be allocated to a large project in the initial stages of the project • Architecture becomes the ‘Bible’ of the development team – whatever they do (or do not do !) is influenced by it. It helps enforce a common design style in all modules. It also helps prevent common errors that might otherwise happen if all modules do their design independently !

21. Project Size Vs. Effort

22. Reduce Software Size • The less software we write, the better it is for project management and for product quality • The cost of software is not just in the cost of ‘coding’ alone; it is also in • Analysis of requirements • Design • Review of requirements, design and code • Test Planning and preparation • Testing • Bug fix • Regression testing

23. Reduce… • ‘Coding’ takes around 15% of development cost • Clearly, if we reduce 15 hrs of coding, we can directly reduce 100 hrs of development effort, and also reduce the project team size appropriately !

24. Reduce… • Size reduction is defined in terms of human-generated source code. Most often, this might still mean that the computer-generated executable code is at least the same or even more • Software Size could be reduced by • Software Re-use • Use of COTS • Programming Languages

25. Software Re-use • Common architectures, common processes, precedent experience, and common environment are all instances of re-use • Black-box re-use refers to component re-use without any internal modification • While-box re-use refers to re-use where some internal changes are necessary before the component could be incorporated • Components get re-used for economic benefits. However, the cost of building a component for re-use could be higher compared to just building it for its intended usage

26. Use of COTS • COTS (Commercial Off-The Shelf Software) are increasingly available to shorten the product development lead time. • Advantages: available much early in the lifecycle, most often good technical support, predictable license cost, mature technology, test software • Disadvantages: frequent upgrades, dependency on (single) vendor, some generalizations might not be very efficient, no control over upgrades and maintenance, extra features that are unnecessary can consume extra resources, etc.

27. Programming Languages • The high-level languages are more ‘expressive’ than the low-level languages – they can ‘express’ the same things with less number of instructions • Typically, what you achieve in C could be achieved with just around 50% SLOC in C++, and around 25% SLOC of VB. Plus, there could be additional savings in the effort for review, rework, etc. • However, not all languages are suitable for all kind of domains. Also, this freedom of language might not always be available

28. Managing Late Changes to Requirements

29. Managing Late Changes… • Rob’s Rule of Large Projects: For Large Projects, there is no such thing as a small change. • Say ‘NO’ when you have to ! • Ideally, the Project Manager should be the gatekeeper of the Change Management process

30. Staffing Large Teams • “…It is impossible to staff a non-trivial project with personnel who all have optimal experience, are fully trained in the tools and technologies, and possess IQs greater than 130.” (page 43, ref 1) • “Teamwork is much more important than the sum of the individuals. With software teams, a project manager needs to configure a balance of solid talent with highly skilled people in the leverage positions.” (page 43, ref 1)

31. Staffing… • Staffing principles from Barry Boehm: • The principle of top talent: use better and fewer people • The principle of job matching: fit the tasks to the skills and motivation of the people available • The principle of career progression: an organization does best in the long run by helping its people to self-actualize • The principle of team balance: select people who will complement and harmonize with one another • The principle of phase-out: keeping a misfit on the team doesn’t benefit anyone

32. Staffing… • In general, staffing is achieved by these common methods: • If people are already available with required skill set, just take them • If people are already available but do not have the required skills, re-train them • If people are not available, recruit trained people • If you are not able to recruit skilled people, recruit and train people

33. Staffing… • Staffing of key personnel is very important: • Project Manager • Software Architect

34. Project Manager • Hiring Skills: Be able to get the right person for the right job • Customer-interface Skills: Avoiding adversarial relationships among stakeholders is a pre-requisite for success • Decision-making skill: Hard to define, this is the most important difference between an average manager and a good manager

35. Project Manager… • Team-building skill: Teamwork requires that a manager establish trust, motivate progress, exploit eccentric prima donnas, transition average people into top performers, eliminate misfits, and consolidate diverse opinions into a team direction • Selling Skill: Must be able to sell all stakeholder of a project for the ultimate success of his project

36. Software Architect • Technical Skills: the most important skills for an architect. These must include skills in both, the problem domain and the solution domain • People Management Skills: must ensure that all people understand and implement the architecture in exactly the way he has conceptualized it. This calls for a lot of people management skills and patience. • Role Model: must be a role model for the software engineers – they would emulate all good (and also all bad !) things that the architect does

37. Recruitment

38. Recruitment • Seven golden rules of recruitment: • Recruit only if you can not get similar talent from within the company • Never hire in a hurry • Always hire people smarter than you • Hire for attitude, train for skills • If you have any doubt, better don’t hire • Hire people who dare to disagree with you • Hire freshers from the college (~15%

39. Recruitment… • Nowadays, hiring a mutual process – even the candidate ‘selects’ the company he wants to work with – so make sure that the interview process is not like an ‘examination’ • Look for the following while hiring • Merit – means the past performance • Potential – means the capability to achieve results in the future • Ambition – what does the person what to achieve in career • For junior people, merit is more important, while for senior people, potential is more important (on a relative scale) • Watch out for fake / clichéd ambition, but never hire someone with low ambition

40. Recruitment… • While hiring, try to hire senior people first – senior people are required for building the team, and junior people can always be in-sourced / staffed during the later phases • While hiring, always hire the people who are ‘star performers’ in their current organizations – it would inspire other good people to join you

41. Training & Mentoring • Not all the people who join your team might be experts, or even have above-average knowledge of your projects’ technology – most of them would need some training or other • Plan for the training upfront • Mentoring ensures a good ‘hand-holding’ for newcomers to a team and culture-building in the team

42. Role Identification • The role must be identified depending on these two factors: • Individual’s capability and interest must be kept in mind as much as possible • Interest of the Project / Organization is always superior to the individual’s interest and capability – in case of a conflict, the larger interest must be honored

43. Task Allocation • Ideally, each person must be able to choose his / her task – this would motivate him / her to give best performance • But, this might not be possible always due to • Some team members joining the project late have less choice • Some team members asking for allocation of task but they are not competent to perform it • Some task is already identified for a more competent person who is yet to join

44. Task… • If the ideal task allocation is not possible, the PM should try to find the next best option / choice and allocate. • If no option is available and the team member does not want to accept an alternate even in the interest of the project / organization, it is better not to take such person in the team – because he might never be a motivated team member.

45. Managing Cross-cultural Teams • Big subject – many books have been written about this • In today’s context, the person who can manage in different cultural work styles is needed. It is not important if you personally subscribe to that way of working or not ! • Some personal experiences: • European: meticulous, think of long-term plan / impact, very high productivity but not keen to overtime, high focus on quality of life and family • Chinese: go-getter attitude, emphasis is on hard work (rather than on smart work), consensus-oriented management,

46. Cross-cultural... • To me, the three most important things are: • Ensure that there is no difference between the expectations and evaluation criteria of tasks between the people of different cultures just because of the culture difference alone • Focus on strengths of different cultures / people – not on its weaknesses • Treat everyone the same

47. Reducing Team Size • Assuming that we have already reduced the software size as much as possible, the primary techniques for reducing the team size are: • Outsourcing • In-sourcing • Improving Productivity • Increasing the project schedule • Making multiple releases

48. Outsourcing • Outsourcing could be an effective way to quickly build a large team, and / or manage peaks and valleys in the project lifecycle • Personally, I think we should plan to outsource 15% - 20% of work to de-risk the project • Make sure you are not putting all your eggs in a single basket !

49. In-sourcing • In-sourcing helps get the people only in the phase when you require, and then reduce them when you have less workload. • I have worked with in-sourced team members with mixed results. The success rates were almost always ~50% • Don’t put too many of in-sourced people in a single team

50. Productivity Metrics ?