Engineering Systems at USC: Center for Systems and Software Engineering

1. Engineering Systems at USC:Center for Systems and Software Engineering Barry Boehm CESUN University Roundtable April 2, 2008 boehm@usc.edu; http://csse.usc.edu

2. 04-02-2008 �USC-CSE 2 Outline CSE, SAE, and CSSE CSSE scope, objectives, and strategy Research and education examples Recent major developments Integrating systems and software engineering Value-based systems and software engineering

3. CSE, SAE, and CSSE Center for Software Engineering (CSE) Large research program, Affiliates� program MSCS-SwE and PhD program Systems Architecting and Engineering (SAE) Program Large MS program Plans for PhD, Affiliates� program Proposal to Affiliates Set up SAE Affiliates� program Membership: $25K/year for either; $40K/year for both Affiliates� response We�re integrating SysE and SwE; you should do this too Result: Center for Systems and Software Engineering Membership: $35K/year ($5K/year for small organizations) 04-02-2008 �USC-CSE 3

4. 10-08-2007 �USC-CSE 4

5. 10-08-2007 �USC-CSE 5 Why Software Projects Fail

6. 10-08-2007 �USC-CSE 6

7. 2007-08 Software Engineering Projects 10-08-2007 �USC-CSE 7

8. 4/18/06 �USC-CSE 8 First Six Weeks of Project Course

9. CSSE Principals (37) Core in CS, SAE, ISE (20) Engineering: Aerospace, Civil, Electrical, Mechanical (8) Schools of Business, Public Policy (4) Information Sciences Institute, Institute for Creative Technology, CREATE Center (5) 7 NAE members; 4 INCOSE Fellows; 9 IEEE Fellows 10-08-2007 �USC-CSE 9

10. Major Research Projects Integrating systems and software engineering (OSD-SSE) 2007: Inhibitors and enablers study 2008: IS&SE via Incremental Commitment Model guidebook Systems of systems engineering (Army FCS, OSD-SSE) 2001- : Future Combat Systems S&SE support 2007- : OSD-SSE Systems of Systems Guidebook support 2007: DACS system of systems cost estimation framework Next-generation space systems (DARPA, NASA) 2003- : Software reliability; processes; risk assessment Scaling up agile methods (Affiliates) 2003- : Architecture-based agile methods Value-based science of design (NSF) 2003- : Value-based theories of software, systems engineering 04-02-2008 �USC-CSE 10

11. 08/31/06 �USC-CSE 11 FCS WinWin Spiral Model

12. 10-08-2007 �USC-CSE 12 How much Architecting is Enough?- A COCOMOII Analysis

13. 10-08-2007 �USC-CSE 13 Tradeoffs Among Cost, Schedule, and Reliability:Want $5.5M, 20 months, 10K hours MTBF

14. 08/31/06 �USC-CSE 14 COSYSMO Operational Concept

15. 04-02-2008 �USC-CSE 15 Outline CSE, SAE, and CSSE CSSE scope, objectives, and strategy Research and education examples Recent major developments Integrating systems and software engineering Value-based systems and software engineering

16. 03/19/2008 �USC-CSSE 16 2007 OSD-SSE Study Scope and Context Statement of Work Evaluate current DoD SysE, SwE processes and standards Primarily SEP Guide, DAG Ch. 4 (SysE), DoDI 5000.2 Identify deltas, issue areas, recommendations Evaluate ICM for integration applicability Evaluate relevant DoD guidance, education, and training Provide recommendations based on evaluations Context: current and future DoD S&SE challenges Multi-owner, multi-mission systems of systems Rapid pace of change Always-on, never-fail systems Need to turn within adversaries� OODA loop Observe, orient, decide, act Within asymmetric adversary constraints The Statement of Work of the 2007 study included systems and software engineering support of efforts by the Office of the Deputy Under Secretary of Defense for Acquisition and Technology (ODUSD(A&T)) Software Engineering and System Assurance (SSA) organization to address issues and gaps identified at the 2006 Defense Software Strategy Summit and in the 2006 NDIA Top Software Issues Workshop. Its direct tasks were to: 1. Evaluate DoD systems engineering lifecycle processes and standards against typical software engineering processes and standards. Prepare a report that delineates any deltas, potential issue areas, and specific recommendations for integrating systems and software engineering.������� 2. Evaluate the Incremental Commitment Model to determine its application to integration of systems engineering, software engineering, and system acquisition life cycles for DoD acquisitions. 3. Evaluate relevant DoD guidance, education, and training and provide recommendations to incorporate findings from tasks 1 and 2. Its sponsor was Mrs. Kristen Baldwin, ODUSD (AT&L)/SSE. Her Technical Representative was Dr. Arthur Pyster, ODUSD(AT&L)/SSE, who was a co-author of the report. Its Principal Investigator (PI) and co-PI have been Dr. Barry Boehm and Ms. JoAnn Lane. Dr. Elliot Axelband, Mr. A. Winsor Brown, Dr. George Friedman, and Dr. Stan Settles, USC have formed its Senior Review Board. Additional USC participants have been Dr. Ray Madachy and Mr. Dan Ingold. The Statement of Work of the 2007 study included systems and software engineering support of efforts by the Office of the Deputy Under Secretary of Defense for Acquisition and Technology (ODUSD(A&T)) Software Engineering and System Assurance (SSA) organization to address issues and gaps identified at the 2006 Defense Software Strategy Summit and in the 2006 NDIA Top Software Issues Workshop. Its direct tasks were to: 1. Evaluate DoD systems engineering lifecycle processes and standards against typical software engineering processes and standards. Prepare a report that delineates any deltas, potential issue areas, and specific recommendations for integrating systems and software engineering.������� 2. Evaluate the Incremental Commitment Model to determine its application to integration of systems engineering, software engineering, and system acquisition life cycles for DoD acquisitions. 3. Evaluate relevant DoD guidance, education, and training and provide recommendations to incorporate findings from tasks 1 and 2. Its sponsor was Mrs. Kristen Baldwin, ODUSD (AT&L)/SSE. Her Technical Representative was Dr. Arthur Pyster, ODUSD(AT&L)/SSE, who was a co-author of the report. Its Principal Investigator (PI) and co-PI have been Dr. Barry Boehm and Ms. JoAnn Lane. Dr. Elliot Axelband, Mr. A. Winsor Brown, Dr. George Friedman, and Dr. Stan Settles, USC have formed its Senior Review Board. Additional USC participants have been Dr. Ray Madachy and Mr. Dan Ingold.

17. An omniscient authority pre-establishes the requirements, preferred system concept, etc. (A4.1, 4.2, 4.4) Technical solutions are mapped and verified with respect to these (A4.1, 4.2, 4.4) They and technology do not change very often (A4.2, 4.5, 6.2) Emphasis on rigor vs. adaptability The program has stable and controllable external interfaces (A4.5) Project organization is function-hierarchical and WBS-oriented (A3.1, 3.2) All requirements are equally important (A4.2) Reviews event/product-based vs. evidence-based (A5.2) The program�s critical path can be defined up front (A6.1) Can achieve optimum readiness at minimum life-cycle cost (A6.5) No slack for contingencies 03/19/2008 �USC-CSSE 17 Current SysE Guidance: Outdated Assumptions Example: SysE Plan Preparation GuideSometimes OK for hardware; generally not for software The SEP guidelines make a number of implicit assumptions which are increasingly invalid as requirements become more emergent than prespecifiable, as requirements and technology undergo more rapid change, and as capabilities and key performance parameters become at least as success-critical as functions and products. Even before Milestone A, Section 4 assumes that requirements are in place as a basis for traceability, verification, and allocation, even before key technologies and COTS products are fully evaluated. This tends to encourage premature specification of requirements, some of which will be found later to be infeasible or inappropriate. Project organization and Integrated Product Teams (IPTs) are assumed to follow functional and Work Breakdown Structure (WBS) hierarchies. As discussed in charts 8 and 9, functional hierarchies are incompatible with layered software structures and inhibit integration of systems and software engineering. IPTs are also needed for non-functional cross-cutting issues such as safety, security, and end-to-end performance. Event- and product-based reviews are better than schedule-based reviews, but they frequently overfocus on presenting functional diagrams without providing any evidence that a product built to satisfy the functions would have adequate performance over a range of stressing scenarios. Evidence of feasibility should also be produced, evaluated, and reviewed for adequacy.The SEP guidelines make a number of implicit assumptions which are increasingly invalid as requirements become more emergent than prespecifiable, as requirements and technology undergo more rapid change, and as capabilities and key performance parameters become at least as success-critical as functions and products. Even before Milestone A, Section 4 assumes that requirements are in place as a basis for traceability, verification, and allocation, even before key technologies and COTS products are fully evaluated. This tends to encourage premature specification of requirements, some of which will be found later to be infeasible or inappropriate. Project organization and Integrated Product Teams (IPTs) are assumed to follow functional and Work Breakdown Structure (WBS) hierarchies. As discussed in charts 8 and 9, functional hierarchies are incompatible with layered software structures and inhibit integration of systems and software engineering. IPTs are also needed for non-functional cross-cutting issues such as safety, security, and end-to-end performance. Event- and product-based reviews are better than schedule-based reviews, but they frequently overfocus on presenting functional diagrams without providing any evidence that a product built to satisfy the functions would have adequate performance over a range of stressing scenarios. Evidence of feasibility should also be produced, evaluated, and reviewed for adequacy.

18. 03/19/2008 �USC-CSSE 18 System/Software Architecture Mismatches- Maier, 2006 System Hierarchy Part-of relationships; no shared parts Function-centric; single data dictionary Interface dataflows Static functional-physical allocation Software Hierarchy Uses relationships; layered multi-access Data-centric; class-object data relations Interface protocols; concurrency challenges Dynamic functional-physical migration A valuable perspective on the mismatches between traditional hardware-oriented systems engineering architectural structures and modern software architectural structures has been provided in [Maier, 2006]. First, traditional systems engineering methods functionally decompose the systems architecture into one-to-many �part-of� or �owned-by� relationships, while modern software methods organize system capabilities as layers of many-to-many service-oriented relationships. An example of the difficulties this causes is given in the next chart. Second, traditional systems engineering approaches collect data into a single data dictionary and store it for common use. This was OK for sequential software, but for interrupt-driven concurrent software, it could cause data being used by a program to be modified while the program was interrupted, causing losses in software integrity. Modern object-oriented software approaches address this problem by using �information-hiding� techniques to store local data inside objects, requiring more up-front data engineering. Third, hardware interfaces tend to be static: sensor data flows down a wire, and the sensor-system interface can be represented by its message content, indicating the data�s form, format, units of measure, precision, frequency, etc. In a net-centric world, interfaces are much more dynamic: a sensor entering a network must go through a number of protocols to register its presence, perform security handshakes, publish and subscribe, etc. The next chart describes how this can result in integration problems. Fourth, hardware relations are assumed to be static and subject to static functional-physical allocation: if the engines on one wing fail, an engine cannot migrate from the other wing to rebalance the propulsion. But in software, modules frequently migrate from one processor to another to compensate for processor failures or processing overloads.A valuable perspective on the mismatches between traditional hardware-oriented systems engineering architectural structures and modern software architectural structures has been provided in [Maier, 2006]. First, traditional systems engineering methods functionally decompose the systems architecture into one-to-many �part-of� or �owned-by� relationships, while modern software methods organize system capabilities as layers of many-to-many service-oriented relationships. An example of the difficulties this causes is given in the next chart. Second, traditional systems engineering approaches collect data into a single data dictionary and store it for common use. This was OK for sequential software, but for interrupt-driven concurrent software, it could cause data being used by a program to be modified while the program was interrupted, causing losses in software integrity. Modern object-oriented software approaches address this problem by using �information-hiding� techniques to store local data inside objects, requiring more up-front data engineering. Third, hardware interfaces tend to be static: sensor data flows down a wire, and the sensor-system interface can be represented by its message content, indicating the data�s form, format, units of measure, precision, frequency, etc. In a net-centric world, interfaces are much more dynamic: a sensor entering a network must go through a number of protocols to register its presence, perform security handshakes, publish and subscribe, etc. The next chart describes how this can result in integration problems. Fourth, hardware relations are assumed to be static and subject to static functional-physical allocation: if the engines on one wing fail, an engine cannot migrate from the other wing to rebalance the propulsion. But in software, modules frequently migrate from one processor to another to compensate for processor failures or processing overloads.

19. 03/19/2008 �USC-CSSE 19 Fractionated, incompatible sensor data management �Touch Football� interface definition earned value Full earned value taken for defining interface dataflow No earned value left for defining interface dynamics Joining/leaving network, publish-subscribe, interrupt handling, security protocols, exception handling, mode transitions Result: all green EVMS turns red in integration Examples of Architecture Mismatches A frequent consequence of traditional systems engineering methods that functionally decompose the systems architecture into one-to-many �part-of� or �owned-by� relationships is to create incompatible and hard-to-modify software. This is exacerbated by the owned-by relations being propagated into work breakdown structures (WBS) and WBS-oriented management structures. For example, an aircraft has wings as parts, which have ailerons as parts, which have aileron controls as parts, which have sensors as parts, which have sensor software as parts. Further, an aircraft using active controls to stabilize an otherwise unstable aircraft will rely on atmospheric, propulsion and attitude sensors owned by other parts of the aircraft hierarchy to accomplish the stabilization objectives. In such situations, best-of-breed hardware subsystem contractors are often selected whose sensor data management software is incompatible, causing major problems as compared to layered software providing unified data management services. And if sensor data management changes need to be coordinated across a wide variety of subsystem management chains, very slow OODA loops result. A recent program�s addressal of this problem is shown in the next chart. A frequent consequence of static interface definitions represented by their message content is that system engineering will credit itself with full interface-definition earned value when it defines the interface message and its content. If no further earned value can be generated for defining the dynamic software interface aspects, they will tend to lose management priority and be deferred until the deficiencies are found during integration, when numerous incompatible subsystem software interface assumptions are found that are expensive and time-consuming to fix. This is a frequent explanation for all-green earned value management system (EVMS) indicators turning red during integration. A frequent consequence of traditional systems engineering methods that functionally decompose the systems architecture into one-to-many �part-of� or �owned-by� relationships is to create incompatible and hard-to-modify software. This is exacerbated by the owned-by relations being propagated into work breakdown structures (WBS) and WBS-oriented management structures. For example, an aircraft has wings as parts, which have ailerons as parts, which have aileron controls as parts, which have sensors as parts, which have sensor software as parts. Further, an aircraft using active controls to stabilize an otherwise unstable aircraft will rely on atmospheric, propulsion and attitude sensors owned by other parts of the aircraft hierarchy to accomplish the stabilization objectives. In such situations, best-of-breed hardware subsystem contractors are often selected whose sensor data management software is incompatible, causing major problems as compared to layered software providing unified data management services. And if sensor data management changes need to be coordinated across a wide variety of subsystem management chains, very slow OODA loops result. A recent program�s addressal of this problem is shown in the next chart. A frequent consequence of static interface definitions represented by their message content is that system engineering will credit itself with full interface-definition earned value when it defines the interface message and its content. If no further earned value can be generated for defining the dynamic software interface aspects, they will tend to lose management priority and be deferred until the deficiencies are found during integration, when numerous incompatible subsystem software interface assumptions are found that are expensive and time-consuming to fix. This is a frequent explanation for all-green earned value management system (EVMS) indicators turning red during integration.

20. 03/19/2008 �USC-CSSE 20 The Incremental Commitment Life Cycle Process: Overview The Incremental Commitment Life Cycle Process: Overview This slide shows how the ICM spans the life cycle process from concept exploration to operations. Each phase culminates with an anchor point milestone review. At each anchor point, there are 4 options, based on the assessed risk of the proposed system. Some options involve go-backs. These options result in many possible process paths. The life cycle is divided into two stages: Stage I of the ICM (Definition) has 3 decision nodes with 4 options/node, culminating with incremental development in Stage II (Development and Operations). Stage II has an additional 2 decision nodes, again with 4 options/node. One can use ICM risk patterns to generate frequently-used processes with confidence that they fit the situation. Initial risk patterns can generally be determined in the Exploration phase. One then proceeds with development as a proposed plan with risk-based evidence at the VCR milestone, adjusting in later phases as necessary. For complex systems, a result of the Exploration phase would be the Prototyping and Competition Plan discussed above. Risks associated with the system drive the life cycle process. Information about the risk(s) (feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the program. A more detailed description of the activities going on in each phase is provided in the final backup chart. The Incremental Commitment Life Cycle Process: Overview This slide shows how the ICM spans the life cycle process from concept exploration to operations. Each phase culminates with an anchor point milestone review. At each anchor point, there are 4 options, based on the assessed risk of the proposed system. Some options involve go-backs. These options result in many possible process paths. The life cycle is divided into two stages: Stage I of the ICM (Definition) has 3 decision nodes with 4 options/node, culminating with incremental development in Stage II (Development and Operations). Stage II has an additional 2 decision nodes, again with 4 options/node. One can use ICM risk patterns to generate frequently-used processes with confidence that they fit the situation. Initial risk patterns can generally be determined in the Exploration phase. One then proceeds with development as a proposed plan with risk-based evidence at the VCR milestone, adjusting in later phases as necessary. For complex systems, a result of the Exploration phase would be the Prototyping and Competition Plan discussed above. Risks associated with the system drive the life cycle process. Information about the risk(s) (feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the program. A more detailed description of the activities going on in each phase is provided in the final backup chart.

21. 03/19/2008 �USC-CSSE 21 Anchor Point Feasibility Evidence Description Evidence provided by developer and validated by independent experts that: If the system is built to the specified architecture, it will Satisfy the requirements: capability, interfaces, level of service, and evolution Support the operational concept Be buildable within the budgets and schedules in the plan Generate a viable return on investment Generate satisfactory outcomes for all of the success-critical stakeholders All major risks resolved or covered by risk management plans Serves as basis for stakeholders� commitment to proceed Anchor Point Feasibility Evidence Description To make ICM concurrency work, the anchor point milestone reviews are the mechanism by which the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these anchor point milestone reviews is focused on developer-produced evidence, documented in a Feasibility Evidence Description (FED), to help the key stakeholders determine the next level of commitment. At each program milestone/anchor point, feasibility assessments and the associated evidence are reviewed and serve as the basis for the stakeholders� commitment to proceed. The FED is not just a document, a set of PowerPoint charts, or Unified Modeling Language (UML) diagrams. It is based on evidence from simulations, models, or experiments with planned technologies and detailed analysis of development approaches and projected productivity rates. The detailed analysis is often based on historical data showing reuse realizations, software size estimation accuracy, and actual developer productivity rates. It is often not possible to fully resolve all risks at a given point in the development cycle, but known, unresolved risks need to be identified and covered by risk management plans.Anchor Point Feasibility Evidence Description To make ICM concurrency work, the anchor point milestone reviews are the mechanism by which the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these anchor point milestone reviews is focused on developer-produced evidence, documented in a Feasibility Evidence Description (FED), to help the key stakeholders determine the next level of commitment. At each program milestone/anchor point, feasibility assessments and the associated evidence are reviewed and serve as the basis for the stakeholders� commitment to proceed. The FED is not just a document, a set of PowerPoint charts, or Unified Modeling Language (UML) diagrams. It is based on evidence from simulations, models, or experiments with planned technologies and detailed analysis of development approaches and projected productivity rates. The detailed analysis is often based on historical data showing reuse realizations, software size estimation accuracy, and actual developer productivity rates. It is often not possible to fully resolve all risks at a given point in the development cycle, but known, unresolved risks need to be identified and covered by risk management plans.

22. 03/19/2008 �USC-CSSE 22 Incremental Commitment in Gambling Total Commitment: Roulette Put your chips on a number E.g., a value of a key performance parameter Wait and see if you win or lose Incremental Commitment: Poker, Blackjack Put some chips in See your cards, some of others� cards Decide whether, how much to commit to proceed Incremental Commitment in Gambling A simple metaphor to help understand the ICM is to compare ICM and incremental-commitment gambling games such as poker and blackjack, to single-commitment gambling games such as Roulette. Many system development contracts operate like Roulette, in which a full set of requirements is specified up front, the full set of resources is committed to an essentially fixed-price contract, and one waits to see if the bet was a good one or not. With the ICM, one places a smaller bet to see whether the prospects of a win are good or not, and decides whether to increase the bet based on better information about the prospects of success.Incremental Commitment in Gambling A simple metaphor to help understand the ICM is to compare ICM and incremental-commitment gambling games such as poker and blackjack, to single-commitment gambling games such as Roulette. Many system development contracts operate like Roulette, in which a full set of requirements is specified up front, the full set of resources is committed to an essentially fixed-price contract, and one waits to see if the bet was a good one or not. With the ICM, one places a smaller bet to see whether the prospects of a win are good or not, and decides whether to increase the bet based on better information about the prospects of success.

23. 03/19/2008 �USC-CSSE 23 Scalable remotely controlled operations Scalable remotely controlled operations � ICM Case Study An example to illustrate ICM benefits is the Unmanned Aerial Vehicle (UAV) (or Remotely Piloted Vehicles (RPV)) system enhancement discussed in Chapter 5 of the NRC HSI report [Pew and Mavor, 2007]. The RPVs are airplanes or helicopters operated remotely by humans. These systems are designed to keep humans out of harm�s way. However, current RPV systems are human-intensive, often requiring two people to operate a single vehicle. If there is a strong desire to modify the 2:1 (2 people to one vehicle) ratio to allow for a single operator and 4 aircraft (e.g., a 1:4 ratio), based on a proof-of principle agent-based prototype demo showing 1:4 performance of some RPV tasks, how should one proceed?Scalable remotely controlled operations � ICM Case Study An example to illustrate ICM benefits is the Unmanned Aerial Vehicle (UAV) (or Remotely Piloted Vehicles (RPV)) system enhancement discussed in Chapter 5 of the NRC HSI report [Pew and Mavor, 2007]. The RPVs are airplanes or helicopters operated remotely by humans. These systems are designed to keep humans out of harm�s way. However, current RPV systems are human-intensive, often requiring two people to operate a single vehicle. If there is a strong desire to modify the 2:1 (2 people to one vehicle) ratio to allow for a single operator and 4 aircraft (e.g., a 1:4 ratio), based on a proof-of principle agent-based prototype demo showing 1:4 performance of some RPV tasks, how should one proceed?

24. 03/19/2008 �USC-CSSE 24 Total vs. Incremental Commitment � 4:1 RPV Total Commitment Agent technology demo and PR: Can do 4:1 for $1B Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months PDR: many outstanding risks, undefined interfaces $800M, 40 months: �halfway� through integration and test 1:1 IOC after $3B, 80 months Incremental Commitment [number of competing teams] $25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not 4:1 $75M, 8 mo. to ACR [3]: agent technology may do 1:1; some risks $225M, 10 mo. to DCR [2]: validated architecture, high-risk elements $675M, 18 mo. to IOC [1]: viable 1:1 capability 1:1 IOC after $1B, 42 months Total vs. Incremental Commitment -- 4:1 RPV This slide outlines two approaches to the RPV question: total commitment and incremental commitment. While this is a hypothetical case for developing a solution to the RPV manning problem, it shows how a premature total commitment without significant modeling, analysis, and feasibility assessment will often lead to large overruns in costs and schedule, and a manning ratio that is considerably less than initially desired. However, by �buying information� early and validating high-risk elements, technologically viable options are identified much earlier and can be provided for a much lower cost and much closer to the desired date. The ICM approach leads to the same improved manning ratio as the total commitment approach, but sooner and at a much reduced cost. The ICM approach also employs a competitive downselect strategy, which both reduces risk and enables a buildup of trust among the acquirers, developers, and users, as emphasized in the Young memo.Total vs. Incremental Commitment -- 4:1 RPV This slide outlines two approaches to the RPV question: total commitment and incremental commitment. While this is a hypothetical case for developing a solution to the RPV manning problem, it shows how a premature total commitment without significant modeling, analysis, and feasibility assessment will often lead to large overruns in costs and schedule, and a manning ratio that is considerably less than initially desired. However, by �buying information� early and validating high-risk elements, technologically viable options are identified much earlier and can be provided for a much lower cost and much closer to the desired date. The ICM approach leads to the same improved manning ratio as the total commitment approach, but sooner and at a much reduced cost. The ICM approach also employs a competitive downselect strategy, which both reduces risk and enables a buildup of trust among the acquirers, developers, and users, as emphasized in the Young memo.

25. 03/19/2008 �USC-CSSE 25 The Incremental Commitment Life Cycle Process: Overview The Incremental Commitment Life Cycle Process: More on the Overview Stage II of the Incremental Commitment Life Cycle provides a framework for concurrent engineering and development of multiple increments. More on this concurrency follows on the next slides. Note: The term �concurrent engineering� fell into disfavor when behind-schedule developers applied it to the practice of proceeding into development while the designers worked on finishing the design. Not surprisingly, the developers encountered a high rework penalty for going into development with weak architecture and risk resolution. �Concurrent engineering� as applied in the ICM is much different. It is focused on doing a cost-effective job of architecture and risk resolution in Stage I; and on performing stabilized development, verification, and validation of the current system increment while concurrently handling the systems change traffic and preparing a feasibility-validated architecture and set of plans for the next increment in Stage II. The Incremental Commitment Life Cycle Process: More on the Overview Stage II of the Incremental Commitment Life Cycle provides a framework for concurrent engineering and development of multiple increments. More on this concurrency follows on the next slides. Note: The term �concurrent engineering� fell into disfavor when behind-schedule developers applied it to the practice of proceeding into development while the designers worked on finishing the design. Not surprisingly, the developers encountered a high rework penalty for going into development with weak architecture and risk resolution. �Concurrent engineering� as applied in the ICM is much different. It is focused on doing a cost-effective job of architecture and risk resolution in Stage I; and on performing stabilized development, verification, and validation of the current system increment while concurrently handling the systems change traffic and preparing a feasibility-validated architecture and set of plans for the next increment in Stage II.

26. 03/19/2008 �USC-CSSE 26 ICM HSI Levels of Activity for Complex Systems ICM HSI Levels of Activity for Complex Systems As mentioned earlier, with the ICM, a number of system aspects are being concurrently engineered at an increasing level of understanding, definition, and development. The most significant of these aspects are shown in this slide, an extension of a similar view of concurrently engineered software projects developed as part of the RUP (shown in a backup slide). As with the RUP version, it should be emphasized that the magnitude and shape of the levels of effort will be risk-driven and likely to vary from project to project. In particular, they are likely to have mini risk/opportunity-driven peaks and valleys, rather than the smooth curves shown for simplicity in this slide. The main intent of this view is to emphasize the necessary concurrency of the primary success-critical activities shown as rows. Thus, in interpreting the Exploration column, although system scoping is the primary objective of the Exploration phase, doing it well involves a considerable amount of activity in understanding needs, envisioning opportunities, identifying and reconciling stakeholder goals and objectives, architecting solutions, life cycle planning, evaluation of alternatives, and negotiation of stakeholder commitments. ICM HSI Levels of Activity for Complex Systems As mentioned earlier, with the ICM, a number of system aspects are being concurrently engineered at an increasing level of understanding, definition, and development. The most significant of these aspects are shown in this slide, an extension of a similar view of concurrently engineered software projects developed as part of the RUP (shown in a backup slide). As with the RUP version, it should be emphasized that the magnitude and shape of the levels of effort will be risk-driven and likely to vary from project to project. In particular, they are likely to have mini risk/opportunity-driven peaks and valleys, rather than the smooth curves shown for simplicity in this slide. The main intent of this view is to emphasize the necessary concurrency of the primary success-critical activities shown as rows. Thus, in interpreting the Exploration column, although system scoping is the primary objective of the Exploration phase, doing it well involves a considerable amount of activity in understanding needs, envisioning opportunities, identifying and reconciling stakeholder goals and objectives, architecting solutions, life cycle planning, evaluation of alternatives, and negotiation of stakeholder commitments.

27. 03/19/2008 �USC-CSSE 27 Different Risk Patterns Yield Different Processes Different Risk Patterns Yield Different Processes As illustrated in the four example paths through the Incremental Commitment Model in this slide, the ICM is not a single monolithic one-size-fits-all process model. As with the spiral model, it is a risk-driven process model generator, but the ICM makes it easier to visualize how different risks create different processes. In Example A, a simple business application based on an appropriately-selected Enterprise Resource Planning (ERP) package, there is no need for a Valuation or Foundations activity if there is no risk that the ERP package and its architecture will not cost-effectively support the application. Thus, one could go directly into the Development phase, using an agile method such as a Scrum/Extreme Programming combination would be a good fit. There is no need for Big Design Up Front (BDUF) activities or artifacts because an appropriate architecture is already present in the ERP package. Nor is there a need for heavyweight waterfall or V-model specifications and document reviews. The fact that the risk at the end of the Exploration phase is negligible implies that sufficient risk resolution of the ERP package�s human interface has been done. Example B involves the upgrade of several incompatible legacy applications into a service-oriented web-based system. Here, one could use a sequential waterfall or V-model if the upgrade requirements were stable, and its risks were low. However, if for example the legacy applications� user interfaces were incompatible with each other and with web-based operations, a concurrent risk-driven spiral, waterfall, or V-model that develops and exercise extensive user interface prototypes and generates a Feasibility Evidence Description (described on chart 12) would be preferable. In Example C, the stakeholders may have found during the Valuation phase that their original assumptions about the stakeholders having a clear, shared vision and compatible goals with respect the proposed new system�s concept of operation and its operational roles and responsibilities were optimistic. In such a case, it is better to go back and assure stakeholder value proposition compatibility and feasibility before proceeding, as indicated by the arrow back into the valuation phase. In Example D, it is discovered before entering the Development phase that a superior product has already entered the marketplace, leaving the current product with an infeasible business case. Here, unless a viable business case can be made by adjusting the project�s scope, it is best to discontinue it. It is worth pointing out that it is not necessary to proceed to the next major milestone before terminating a clearly non-viable project, although stakeholder concurrence in termination is essential. Different Risk Patterns Yield Different Processes As illustrated in the four example paths through the Incremental Commitment Model in this slide, the ICM is not a single monolithic one-size-fits-all process model. As with the spiral model, it is a risk-driven process model generator, but the ICM makes it easier to visualize how different risks create different processes. In Example A, a simple business application based on an appropriately-selected Enterprise Resource Planning (ERP) package, there is no need for a Valuation or Foundations activity if there is no risk that the ERP package and its architecture will not cost-effectively support the application. Thus, one could go directly into the Development phase, using an agile method such as a Scrum/Extreme Programming combination would be a good fit. There is no need for Big Design Up Front (BDUF) activities or artifacts because an appropriate architecture is already present in the ERP package. Nor is there a need for heavyweight waterfall or V-model specifications and document reviews. The fact that the risk at the end of the Exploration phase is negligible implies that sufficient risk resolution of the ERP package�s human interface has been done. Example B involves the upgrade of several incompatible legacy applications into a service-oriented web-based system. Here, one could use a sequential waterfall or V-model if the upgrade requirements were stable, and its risks were low. However, if for example the legacy applications� user interfaces were incompatible with each other and with web-based operations, a concurrent risk-driven spiral, waterfall, or V-model that develops and exercise extensive user interface prototypes and generates a Feasibility Evidence Description (described on chart 12) would be preferable. In Example C, the stakeholders may have found during the Valuation phase that their original assumptions about the stakeholders having a clear, shared vision and compatible goals with respect the proposed new system�s concept of operation and its operational roles and responsibilities were optimistic. In such a case, it is better to go back and assure stakeholder value proposition compatibility and feasibility before proceeding, as indicated by the arrow back into the valuation phase. In Example D, it is discovered before entering the Development phase that a superior product has already entered the marketplace, leaving the current product with an infeasible business case. Here, unless a viable business case can be made by adjusting the project�s scope, it is best to discontinue it. It is worth pointing out that it is not necessary to proceed to the next major milestone before terminating a clearly non-viable project, although stakeholder concurrence in termination is essential.

28. 03/19/2008 Common Risk-Driven Special Cases of the ICM Common Risk-Driven Special Cases of the ICM: Because of its complexity, ICM examples have been developed to show users how to use the framework to create a development process appropriate for their system of interest. These cases cover the very small to the very large as well as the use of commercial off-the-shelf (COTS) software products to the development of a large, complex custom software application or integrated sets of software applications. Each of these situations presents risks at the various stages of development, some risks more critical than others. The goal of the ICM is to identify these risks and then tailor the process to include rigor where necessary to investigate and manage the risks and to streamline the process when risks are negligible, allowing the development team to be more agile when possible. This table contains a list of the special cases of the ICM and an example of each case. Each of the special cases is described in further detail in a companion briefing. In general, a project will be able to identify its special case by the end of the Exploration phase.Common Risk-Driven Special Cases of the ICM: Because of its complexity, ICM examples have been developed to show users how to use the framework to create a development process appropriate for their system of interest. These cases cover the very small to the very large as well as the use of commercial off-the-shelf (COTS) software products to the development of a large, complex custom software application or integrated sets of software applications. Each of these situations presents risks at the various stages of development, some risks more critical than others. The goal of the ICM is to identify these risks and then tailor the process to include rigor where necessary to investigate and manage the risks and to streamline the process when risks are negligible, allowing the development team to be more agile when possible. This table contains a list of the special cases of the ICM and an example of each case. Each of the special cases is described in further detail in a companion briefing. In general, a project will be able to identify its special case by the end of the Exploration phase.

29. 03/19/2008 �USC-CSSE 29 Example ICM Commercial Application:Symbiq Medical Infusion PumpWinner of 2006 HFES Best New Design AwardDescribed in NRC HSI Report, Chapter 5 Example ICM HCI Application: Symbiq Medical Infusion Pump During the National Research Council Human-System Integration study, the ICM was refined by analyzing and formalizing examples of best commercial practices of human-system integration. The Symbiq Medical Infusion Pump shown above was developed by Abbott Laboratories and won the Best New Design of 2006 Award from the Human Factors and Ergonomics Society. As evident from the views above, such devices present significant challenges in integrating high-safety.hardware, software, and human factors engineering This next-generation infusion pump is a general-purpose intravenous infusion pump (IV pump) designed primarily for hospital use with secondary, limited-feature use by patients at home. The device is intended to deliver liquid medications, nutrients, blood ,and other solutions at programmed flow rates, volumes, and time intervals via intravenous and other routes to a patient. The device offers medication management features, including medication management safety software through a programmable drug library. The infuser also has sufficient memory to support extensive tracking logs and the ability to communicate and integrate with hospital information systems. The infuser is available as either a single-channel pump or a dual-channel pump. The two configurations can be linked together o form a 3- or 4- channel pump. The infuser includes a large touchscreen color display and can be powered by either A/C power or rechargeable batteries. More detail on its use of the ICM is in the backup charts and in Chapter 5 of [Pew and Mavor, 2007].Example ICM HCI Application: Symbiq Medical Infusion Pump During the National Research Council Human-System Integration study, the ICM was refined by analyzing and formalizing examples of best commercial practices of human-system integration. The Symbiq Medical Infusion Pump shown above was developed by Abbott Laboratories and won the Best New Design of 2006 Award from the Human Factors and Ergonomics Society. As evident from the views above, such devices present significant challenges in integrating high-safety.hardware, software, and human factors engineering This next-generation infusion pump is a general-purpose intravenous infusion pump (IV pump) designed primarily for hospital use with secondary, limited-feature use by patients at home. The device is intended to deliver liquid medications, nutrients, blood ,and other solutions at programmed flow rates, volumes, and time intervals via intravenous and other routes to a patient. The device offers medication management features, including medication management safety software through a programmable drug library. The infuser also has sufficient memory to support extensive tracking logs and the ability to communicate and integrate with hospital information systems. The infuser is available as either a single-channel pump or a dual-channel pump. The two configurations can be linked together o form a 3- or 4- channel pump. The infuser includes a large touchscreen color display and can be powered by either A/C power or rechargeable batteries. More detail on its use of the ICM is in the backup charts and in Chapter 5 of [Pew and Mavor, 2007].

30. 04-02-2008 �USC-CSE 30 Outline CSE, SAE, and CSSE CSSE scope, objectives, and strategy Research and education examples Recent major developments Integrating systems and software engineering Value-based systems and software engineering (VBSEE)

31. 06/25/07 � USC-CSE 31 VBSEE Motivation and Context INCOSE Definition of �Systems Engineering� An interdisciplinary approach and means to enable the realization of successful systems A systems engineering theory should provide criteria and guidance for realizing successful systems Addressing the Intellectual Content of Systems Engineering How distinguished from other engineering disciplines How related to other theories with intellectual content

32. 06/25/07 � USC-CSE 32 Intellectual Content of Systems Engineering: Distinguishing Features The intellectual content of most engineering disciplines is component-oriented and value-neutral Ohm�s Law, Hooke�s Law, Newton�s Laws The intellectual content of systems engineering is focused on: How people and components can be combined into a cost-effective system System definition, architecting, analysis, planning, evaluation, improvement Cost-effectiveness in terms of value to stakeholders The ability to address value considerations is an essential qualification for a theory of systems engineering

33. 06/25/07 � USC-CSE 33 Theory W: Enterprise Success Theorem� And informal proof Theorem: Your enterprise will succeed if and only if it makes winners of your success-critical stakeholders Proof of �if�: Everyone that counts is a winner. Nobody significant is left to complain. Proof of �only if�: Nobody wants to lose.Prospective losers will refuse to participate, or will counterattack.The usual result is lose-lose.

34. 06/25/07 � USC-CSE 34 Theory W: WinWin Achievement Theorem Making winners of your success-critical stakeholders requires: Identifying all of the success-critical stakeholders (SCSs). Understanding how the SCSs want to win. Having the SCSs negotiate a win-win set of product and process plans. Controlling progress toward SCS win-win realization, including adaptation to change.

35. 06/25/07 � USC-CSE 35 VBSET Component Theories Theory W (Stakeholder win-win) Enterprise Success Theorem, Win-Win Achievement Theorem Dependency Theory (Product, process, people interdependencies) Systems architecture/performance theory, costing and scheduling theory; organization theory Utility Theory Utility functions, bounded rationality, Maslow need hierarchy, multi-attribute utility theory Decision Theory Statistical decision theory, game theory, negotiation theory, Theory of Justice Control Theory Observability, predictability, controllability, stability theory

36. 08/2007 �USC-CSSE 36 Initial VBSE Theory: 4+1 Process� With a great deal of concurrency and backtracking

37. 06/25/07 � USC-CSE 37 Value-Based Testing: Empirical Data and ROI� LiGuo Huang, ISESE 2005

38. 04/19/07 38

39. Conclusions: IS&SE with ICM Current DoD (and other) SysE guidance seriously inhibits SwE Largely sequential definition, design, development, integration, and test Slows agility, ability to turn inside adversaries� OODA loop Functional �part-of� vs. layered �served by� product hierarchy Static vs. dynamic interfaces: messages vs. protocols One-size-fits-all process guidance inhibits balancing agility and assurance Incremental Commitment Model (ICM) enables better IS&SE Integrates hardware, software, human factors aspects of SysE Concurrent exploration of needs, opportunities, solutions Enabled by value-based systems engineering theory and process Successfully addresses issues above in commercial practice Only partially proven in DoD practice (examples: CrossTalk Top-5 projects) Key practices applied to help major programs (example: Future Combat Systems) Being adopted by major programs (example: Missile Defense Agency) Effort underway to work with adopters to develop DoD ICM Guidebook, in coordination with other USD(AT&L)SSE initiatives As further elaborated in the backup charts, when systems engineering was initially defining itself, maturing , and setting standards in the 1960s and 1970s, its focus was on hardware artifacts. People were not part of �the system� but users of it. In the 1980s, some pioneers began to experience that systems were getting more software-intensive and human-intensive, and formulated initial approaches such as Soft Systems Engineering [Checkland, 1980] and Systems Architecting [Rechtin, 1991] to integrate these factors into systems engineering. Due to the fact that general changes in standards and guidelines take a good deal of time, the DoD systems engineering standards area is still in the process of assimilating these perspectives. The results of the 2007 IS&SE study and associated results such as the [Maier, 2006] system and software architectures paper and the NDIA Software Summit identification of critical software issues indicate that software engineering best practices are being seriously inhibited by current DoD systems engineering guidance, and that improvement avenues are available in such areas as acquisition, architecture, and processes. The ICM provides an avenue for adapting current best commercial IS&SE practices for use on DoD projects. USC-CSSE�s current effort to work with pilot adopters to develop a DoD ICM Guidebook, in coordination with other USD(AT&L)SSE initiatives, affords opportunities for further candidate pilot adopters to collaborate in this effort. As further elaborated in the backup charts, when systems engineering was initially defining itself, maturing , and setting standards in the 1960s and 1970s, its focus was on hardware artifacts. People were not part of �the system� but users of it. In the 1980s, some pioneers began to experience that systems were getting more software-intensive and human-intensive, and formulated initial approaches such as Soft Systems Engineering [Checkland, 1980] and Systems Architecting [Rechtin, 1991] to integrate these factors into systems engineering. Due to the fact that general changes in standards and guidelines take a good deal of time, the DoD systems engineering standards area is still in the process of assimilating these perspectives. The results of the 2007 IS&SE study and associated results such as the [Maier, 2006] system and software architectures paper and the NDIA Software Summit identification of critical software issues indicate that software engineering best practices are being seriously inhibited by current DoD systems engineering guidance, and that improvement avenues are available in such areas as acquisition, architecture, and processes. The ICM provides an avenue for adapting current best commercial IS&SE practices for use on DoD projects. USC-CSSE�s current effort to work with pilot adopters to develop a DoD ICM Guidebook, in coordination with other USD(AT&L)SSE initiatives, affords opportunities for further candidate pilot adopters to collaborate in this effort.

40. 2008 SOW: DoD ICM Guidebook Develop a draft Guidebook for next-generation DoD IS&SE based on the ICM In collaboration with DoD and industry participants Collaborate with selected DoD and industry parties in experimental tailoring of draft Guidebook elements Hold a series of Guidebook workshops Mar 19-20 (USC), July 15-17 (DC area), Oct 29-30 (USC) Coordinate Guidebook with other IS&SE initiatives Systems of systems engineering, systemic analysis, acquisition, education, assessment, research and technology Task 1.� In collaboration with DoD and industry participants, develop a draft Guidebook for next-generation DoD integrated systems and software engineering (IS&SE) based on the Incremental Commitment Model (ICM).� The guidebook will include:� - General principles, practices, and DoD milestone and SE technical review criteria common to all projects using the ICM;� - Detailed guidelines for using the ICM key practices to integrate systems and software engineering, and to prepare for DoD milestone and SE technical reviews during each DoD acquisition phase;� - Process decision criteria enabling early determination of project-appropriate, risk-driven special cases of the ICM;� - Guidelines for ICM-compatible interpretations of current DoD regulations, specifications, and standards;� - Examples of good and bad interpretations of ICM principles and practices;� - Guidelines for enhancing current projects with selected ICM principles and practices.Task 2.� Collaborate with selected DoD and industry projects in experimental tailoring of early draft Guidebook elements for application to existing, new, and experimental pilot projects.� Candidate projects include the Missile Defense Agency, the Army Future Combat Systems program, a TBD Air Force space program, and a TBD Navy program.Task 3.� Hold a series of government-industry workshops to obtain feedback on evolving drafts of Guidebook material.� Candidate times and places would be in March at USC; in July on the east coast, and in October at USC.Task 4.� Coordinate development of the draft Guidebook with other OSD-AT&L IS&SE initiatives (e.g., acquisition, education, assessment, systemic analysis, systems of systems engineering, research and technology agendas). Task 5.� In particular, coordinate with the SSE software systemic analysis initiative.Task 6.� In particular, coordinate with the SSE systems of systems engineering initiative.Task 1.� In collaboration with DoD and industry participants, develop a draft Guidebook for next-generation DoD integrated systems and software engineering (IS&SE) based on the Incremental Commitment Model (ICM).� The guidebook will include:

41. 03/19/2008 �USC-CSSE 41 References - I Beck, K., Extreme Programming Explained, Addison Wesley, 1999. Boehm, B., �Some Future Trends and Implications for Systems and Software Engineering Processes�, Systems Engineering 9(1), pp. 1-19, 2006. Boehm, B., Brown, W., Basili, V., and Turner, R., �Spiral Acquisition of Software-Intensive Systems of Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004. Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006. Boehm, B., and Lane, J., �Using the ICM to Integrate System Acquisition, Systems Engineering, and Software Engineering,� CrossTalk, October 2007, pp. 4-9. Boehm, B., and Lane, J., �A Process Decision Table for Integrated Systems and Software Engineering,� Proceedings, CSER 2008, April 2008. Boehm, B., �Future Challenges and Rewards for Software Engineers,� DoD Software Tech News, October 2007, pp. 6-12. Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000. Boehm, B., Software Engineering Economics, Prentice Hall, 2000. Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001. Carlock, P., and J. Lane, �System of Systems Enterprise Systems Engineering, the Enterprise Architecture Management Framework, and System of Systems Cost Estimation�, 21st International Forum on COCOMO and Systems/Software Cost Modeling, 2006. Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2nd ed., 1999). Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/, 2006. Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May 2003. Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004. Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006. Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.

42. 03/19/2008 �USC-CSSE 42 References -II Highsmith, J., Adaptive Software Development, Dorset House, 2000. International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC 12207, 1995 ISO, Systems Engineering � System Life Cycle Processes, ISO/IEC 15288, 2002. Jensen, R. �An Improved Macrolevel Software Development Resource Estimation Model,� Proceedings, ISPA 5, April 1983, pp. 88-92. Krygiel, A., Behind the Wizard�s Curtain; CCRP Publication Series, July, 1999, p. 33 Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007. Lane, J. and Valerdi, R., �Synthesizing SoS Concepts for Use in Cost Estimation�, Proceedings of IEEE Systems, Man, and Cybernetics Conference, 2005.� Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980. Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development", USC CSSE Technical Report USC-CSSE-2006-623, 2006. Maier, M., �Architecting Principles for Systems-of-Systems�; Systems Engineering, Vol. 1, No. 4 (pp 267-284). Maier, M., �System and Software Architecture Reconciliation,� Systems Engineering 9 (2), 2006, pp. 146-159. Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute, 2006. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. Putnam, L., �A General Empirical Solution to the Macro Software Sizing and Estimating Problem,� IEEE Trans SW Engr., July 1978, pp. 345-361. Rechtin, E. Systems Architecting, Prentice Hall, 1991. Schroeder, T., �Integrating Systems and Software Engineering: Observations in Practice,� OSD/USC Integrating Systems and Software Engineering Workshop, http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007.

43. 03/19/2008 List of Acronyms B/L Baselined C4ISR Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance CD Concept Development CDR Critical Design Review COTS Commercial Off-the-Shelf DCR Development Commitment Review DI Development Increment DoD Department of Defense ECR Exploration Commitment Review EVMS Earned Value Management System FCR Foundations Commitment Review FDN Foundations, as in FDN Package FED Feasibility Evidence Description FMEA Failure Modes and Effects Analysis FRP Full-Rate Production GAO Government Accountability Office GUI Graphical User Interface

44. 03/19/2008 List of Acronyms (continued) HMI Human-Machine Interface HSI Human-System Interface HW Hardware ICM Incremental Commitment Model IOC Initial Operational Capability IRR Inception Readiness Review IS&SE Integrating Systems and Software Engineering LCA Life Cycle Architecture LCO Life Cycle Objectives LRIP Low-Rate Initial Production MBASE Model-Based Architecting and Software Engineering NDI Non-Developmental Item NRC National Research Council OC Operational Capability OCR Operations Commitment Review OO&D Observe, Orient and Decide OODA Observe, Orient, Decide, Act O&M Operations and Maintenance

45. 03/19/2008 List of Acronyms (continued) PDR Preliminary Design Review PM Program Manager PR Public Relations PRR Product Release Review RUP Rational Unified Process SoS System of Systems SoSE System of Systems Engineering SSE Systems and Software Engineering SW Software SwE Software Engineering SysE Systems Engineering Sys Engr Systems Engineer S&SE Systems and Software Engineering USD (AT&L) Under Secretary of Defense for Acquisition, Technology, and Logistics VCR Validation Commitment Review V&V Verification and Validation WBS Work Breakdown Structure WMI Warfighter-Machine Interface