Is Agility Useful In Military Software Projects?

1. Is Agility Useful In Military Software Projects? Terry Shepard Electrical and Computer Engineering Royal Military College of Canada University of Toronto, 6 July 2007 shepard@rmc.ca 613-541-6000 ext. 6031

2. An Aside:Software Engineering at RMC • 4 civilian profs, 3 military profs • 7 in ECE, 1 in Math & CS • one emeritus prof (as of today!) • undergrad option as part of computer engineering • small classes! • thesis and project pattern master’s degrees • PhD: 3 grads so far (examining 4th one July 13) • may have graduated the first Software Eng’g PhD in Canada (2004)

3. from the Abstract • Three obvious characteristics of [agile] projects are that • a single representative customer can speak with authority on behalf of all stakeholders, • development is a collaborative effort and not adversarial, • the size of the group is small enough that the customer can always be available to any developer with a question

4. ‘a single representative customer can speak with authority on behalf of all stakeholders’ • works when there is a clearly defined user community, with relatively uniform needs, and few unresolved issues among the stakeholders • consider two examples • automobile parts warehouse in Pakistan (case study from a grad student for a requirements course at Queen’s) • Canadian Land Forces command and control system acquisition

5. One Warehouse: simple ? Example 1 At the start: very limited computer use…

6. Warehouse Stakeholders – not so simple! • There were six (6) directly and six (6) indirectly related stakeholders in this warehouse situation: • Inventory Management • Order Placement • Audit • Sales • Management Tier • Invoicing Department • Customers • General Supplier • Parts Manufacturer • Garage • After Market Manufacturers/Refurbishers • Transportation and Delivery

7. Warehouse example illustrates the role of a champion • champion can act as a surrogate customer • gathers information from all the stakeholders • synthesizes the real needs, how to keep everyone happy • champion can then act as a single customer point of contact in an agile development team • role of champion can also be used to capture requirements in more conventional ways

8. Example 2: Canadian Land Forces command and control system acquisition • more stakeholders than the warehouse example • operational commanders in the field are key • needed to field early versions of system in training exercises – another kind of agility • didn’t happen • delivery of whole unusable system in 2000 • portions are starting to be used in Afghanistan • Master’s thesis in progress on the 50 year history of these requirements

9. ‘development is a collaborative effort and not adversarial’ • a major challenge in most military software projects • consider two examples: • military officers embedded in supplier organization while contract is in progress • writing requirements while the contractor is already writing code to their internal requirements

10. Example 1) : military officers embedded in supplier organization while contract is in progress • consider Brook’s Law: “adding more people to a late project makes it later” • even more true if development is agile? • agility might demand that an officer try to stop the people from being added • corporate and military chains of command have to be agile • in the specific case, they were not • hierarchical structure on both sides of the contract can be adapted to agility? • what does agility mean?

11. many senses of agile/agility • narrow sense refers to the ‘Agile Manifesto’ for software development (next slide) • broader sense refers to flexibility in an organization, ability to respond to change, management openness to decisions being questioned • can be a recipe for inaction • military in action tends to be highly agile, less so in peacetime • e.g. land forces doctrine changes normally take months or years, but only weeks or even days in Afghanistan

12. Agile Manifesto “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.”

13. Example 2) : writing requirements while the contractor is already writing code to their internal requirements • it is often the case in military software contracts that there are two or more parallel requirements documents • e.g. CCS 330 command and control system had about 3000 requirements written by DND • parallel 14 volume Combat System Users Manual (CSUM) written by Lockheed Martin • a number of defects reference incompatibility between the two – CSUM tends to reflect reality • CSUM needed for training and operations

14. Halifax Class Frigate – CCS 330(entered service in 1989)

15. recent RMC Master’s thesis on CCS 330 • Exploration Of A Heterogeneous Tracing Strategy To Support Naval Command And Control Software – by Paul Mondoux • retrospective study tracing between known defects and requirements • are tracing and agility compatible? • introducing heterogeneity is a form of agility

16. Mondoux Thesis Abstract Requirements traceability aims to improve project communication, to support dealing with the effects brought upon by change, to preserve system and decision knowledge and to help assure quality. Even though mandated for Naval command and control projects, there remains poor software requirement traceability take-up due to a perceived lack of benefit in its use. Static link based traceability schemes, if not properly conceived and maintained, can create an unmanageable tangle of links, often providing little to no return on the time and money invested. Based on analysis of a subset of the roughly 3000 individual requirements for the Navy’s CCS330 command and control system, this thesis will investigate whether a heterogeneous tracing strategy based on requirement-defect risk analysis may be an alternative to traditional link-based traceability strategies. Attributes considered include probability of defect, degree of impact, standards/mandates, evolution/maintainability of requirements, requirement type (Functional/Non-Functional) and granularity. A successful application of a heterogeneous tracing strategy will simplify tracing by better matching link types to requirement types while minimizing cost and maximizing value as compared to a link based strategy and a no trace strategy.

17. Heterogeneous Tracing [from Mondoux thesis]

18. ‘the size of the group is small enough that the customer can always be available to any developer with a question’ • many project leaders have had very good success scaling this one up • solution seems to be to do some serious planning up front to allocate the work to smaller teams • for example, article by Philippe Kruchten now of UBC, on ‘Scaling down large projects to meet the agile sweet spot’

19. article by Philippe Kruchten (1) • “This initial architecture team proceeds iteratively, and the team gradually grows in size. When this team reaches fifteen members or more, they split into: • The architecture team, mostly customer involved, and making key architectural choices • The prototyping team, gradually building up and testing the architectural prototype, refactoring it, and pushing back up any issues detected, thus closing the loop”

20. article by Philippe Kruchten (2) “ The architecture team (seven to twelve people) is the customer for two or three prototyping teams (consisting of thirty-five or more people). “ Staffing the project: Construction and Transition • Towards the end of the elaboration phase, when the structure of the code starts to take shape and stabilize, additional teams are created, each one "owning" a chunk of the architecture. The implementation view (according to the "4+1 view model" serves as the basis for the partition of the code and the allocation of ownership to a set of teams. Each chunk of code now belongs to one and only one team.”

21. article by Philippe Kruchten (3) “ Ideally, each of the newly created teams is seeded with one or two members of the original architecture and prototyping teams to bring some of the project knowledge and emerging culture. Typically, each of these seed members is the new team's "technical lead." This person will play the role of the local architect and serve as the communication conduit to the core architecture team that remains, and, in some cases, with the customer.”

22. Can military contracts call up this style of agility? • agility is mostly internal to the contractor • telling the contractor how to manage the contract internally causes finger pointing • finding contractors who work this way of their own accord may get easier … • but it’s a very large change for most large aerospace contractors

23. Diversity of Military Software Projects • many shapes and sizes – we will consider a few more … • some officers develop local DB applications privately – they work and are used – good outcome! – probably agile … • e.g. home grown project tracking system • agility in projects requiring safety certification • recent RMC Master’s thesis on agility in the context of DO178B – by Ron Chisholm • very large projects

24. Agility in the Context of DO178B • DO178B is the avionics safety standard enforced by the FAA – 5 levels • four Stages of Involvement by the Certification Authority • Agility is compatible with DO178B • but changes are needed eg. to increase the amount of planning • Agility may bring benefits • e.g. test cases developed with code • e.g. write requirements near end of each iteration, when they are stable and complete

25. A very large project: The Fox is more Agile than the Chickens • US Coast Guard Deepwater project started in 2002 • $24B over 20 years to refurbish whole fleet of ships and planes • handed requirements responsibility to prime contractor, who was also to build the ‘system of systems’ • the prime (fox) can eviscerate systems (chickens), thereby improving its financial nourishment • Puts contractor in position of both specifying what to build, and building it • Coast Guard has now changed the terms of the prime contract

26. Deepwater Planned Assets

27. Deepwater is Software? • Integrated Deepwater System Program, often simply called "Deepwater" or "IDS," is the largest and most innovative acquisition in the Coast Guard's history. Deepwater is not just "new ships and aircraft," but an integrated, performance-based approach to upgrading existing assets while transitioning to newer, more capable platforms, with improved systems for Command, Control, Communications and Computers, Intelligence, Surveillance, Reconnaissance (C4ISR) and innovative logistics support

28. C4ISR

29. “Performance Based Acquisition” “… Deepwater focuses on system-wide capabilities and not assets. The Coast Guard began the design process with the goal to acquire the performance capabilities required to execute the full range of Coast Guard deepwater missions. Traditionally, acquisition programs have replaced a single type of asset - a type of cutter, for example, or class of helicopter - on a one for one basis. The Deepwater Program takes a different approach - a system-of-systems perspective. The Coast Guard is focusing on the overall required capabilities - the entire system’s capabilities and the life cycle costs, including people, equipment, infrastructure, and operations - rather than the individual assets. Performance-based acquisitions are consistent with the Budget and Performance Integration initiative of the President’s Management Agenda. In simple terms, this initiative states that the Government should be results-oriented and should be guided by performance not by process.”

30. Department of Homeland Security (DHS) Office of Inspector General (OIG) Audit released in Aug. 2006 • DHS OIG conducted an audit in Aug. 2006 of “the development and implementation of information technology (IT) systems to support the U.S. Coast Guard’s Integrated Deepwater System program.” • Purpose was “to evaluate the effectiveness of Coast Guard efforts to identify C4ISR systems requirements, and to determine how well these systems are being implemented to support the Deepwater program.”

31. Sample Audit Finding 1 “ agency officials are involved in high-level Deepwater requirements definition processes, they have limited influence over the decisions that the contractor makes toward meeting those IT requirements. Due to a lack of discipline in adhering to systems requirements change management processes, there is little assurance that the requirements are complete or effective in meeting program goals.”

32. Sample Audit Finding 2 “ many IT requirements documents do not require prior Coast Guard approval. For example, according to one contract official, the contractor generally gives the Coast Guard 30 days to review documents. However, because of a shortage of personnel to conduct the document reviews, the agency has difficulty responding within that time frame.”

33. Main Lesson to be learned from Deepwater • Attempted to give contractor maximum flexibility (agility) while reducing oversight • Balance between oversight and contract freedom must be carefully nuanced • Deepwater had too little oversight • Too much oversight can slow progress to a crawl • There are no simple solutions in the management of large projects • ideas of Kruchten and others on large project agility should be applicable

34. Can DND be the fox and the contractors be the chickens? • One tactic: • you have a high level vision of what is needed • put out an RFP to develop requirements • contractors develop their proposals at their own expense • you get a range of approaches to the next level of detail in requirements • contract is let to develop the requirements • not clear how well this scales to large projects • feels agile at the RFP stage, less so once contract is let

35. What’s wrong with the fox and chickens model? • One side wins – the other side loses • fox may kill more chickens than it needs right away • farmer may increase security – fox may starve • Best arrangement is win-win • if any party feels threatened, they will act to protect themselves • the software agility movement essentially promotes win-win • win-win is the ideal – practice is more complex • win-win contracting can be especially difficult

36. Agility, Open Source and Military Software • It may be possible to combine agility with the approach used in the open source community • code base in continuous evolution • all parties share in that evolution • These ideas are being considered at US DoD deputy secretary level

37. Open Technology Development Roadmap Plan April 2006 Prepared for: Ms. Sue Payton Deputy Under Secretary of Defense Advanced Systems & Concepts

38. Open Technology Development – DoD Roadmap Plan “ OSS and open source development methodologies are important to the National Security and National Interest of the U.S. for the following reasons: • Enhances agility of IT industries to more rapidly adapt and change to user needed capabilities. • Strengthens the industrial base by not protecting industry from competition. Makes industry more likely to compete on ideas and execution versus product lock-in. • Adoption recognizes a change in our position with regard to balance of trade of IT. • Enables DoD to secure the infrastructure and increase security by understanding what is actually in the source code of software installed in DoD networks. • Rapidly respond to adversary actions as well as rapid changes in the technology industrial base.”

39. Open Technology Development – DoD Roadmap Plan “ Open Technology Development combines salient advances in the following areas: 1. Open Standards and Interfaces 2. Open Source Software and Designs 3. Collaborative/Distributive culture and the and online support tools 4. Technological Agility”

40. Open Technology Development – DoD Roadmap Plan “ the hinge issues are: • How can DoD leverage military-funded software development more effectively? • How can OTD’s business-process advantages increase both the rate of innovation and the sustainability of software developed using DoD funds? • What changes in acquisition practice and policy may be required to capture the benefit of OTD within and across the Defense Department? • How can DoD leverage existing external OSS resources?”

41. Conclusions • Opportunities exist for agility in military software • Diversity dominates: agility needs to be applied appropriately • Evolution will be slow: there is ponderous machinery to change

42. Requirements Engineering Guide • Goal: describe what needs to be known to be successful in requirements engineering tasks for DND software intensive projects • ambitious: many levels, layers of complexity • To have value, the guide needs to be put into initial use and then revised

43. what people responsible for requirements for military software projects need to know about requirements engineering • what can go wrong • how it can go wrong • things to do to make it go right

44. examples of what can go wrong • too many changes • high costs in adversarial environments • missing requirements • goldplating • wrong granularity • untestable requirements • unsatisfied stakeholders

45. examples of how it can go wrong • wrong people • not enough resources • poor judgment • misunderstanding • fox in charge of the chicken coop

46. things to do to make it go right • tell truth to power • know the whole history • use contracting creatively • find things that will work • know who all the stakeholders are, and find out what they know • iterate: approach the solution gradually • build trust with the contractor • work really hard, but pray!