NDIA Top SE and SW Issues – 2010 Report on NDIA SE and SW Workshops Held March 16-18, 2010

1. NDIA Top SE and SW Issues – 2010 Report on NDIA SE and SW Workshops Held March 16-18, 2010 Report to NDIA SE Division June 15, 2010 Facilitators – SED Vice Chairs: • SW Issues - Geoff Draper, Harris • SE Issues - Hal Wilson, Northrop Grumman

2. Task Description • Identify Top Systems Engineering and Software Engineering problems or issues prevalent within the defense industry • Assess progress and update issues from 2006 workshops • Document issues • Description and current state • Rationale and SW impacts • Develop recommendations (short term and long term) • Generate task report • Submit to OSD by May 2010

3. Task Group Activities Top SE-SW Issues (2006) Top SE/SW Issues(2010) Identify Candidate SW Issues Prioritize SW Issues and Actions Document Top SW Issues for OSD Report Task reports Candidate issues Status of OSD/Industry Initiatives Detailed sub-issues Effects and root causes Consensus review Issues Detailed description Recommendations Background Brainstorming Group discussion NDIA reports on Top SE and SW issues in progress. Due to be submitted to OSD May 2010.

4. NDIA Top Software Issues – 2010 Preliminary Results - NDIA Top SW Issues Workshop March 16-17, 2010

5. 2010 - Common Themes Noted • The need for rapid development has increased – rapid response to mission needs (faster, better, cheaper). Need to find ways to be more responsive, quicker. • The need for better SW architectures and modular components has been reinforced. Architectures are more difficult when accounting for legacy upgrades, SoS, and interoperability. • Agile terminology is often misused and could be misleading Stakeholder commitment to Agile development processes is not widely understood (acquisition agility is not the same as Scrum). • SW community must be engaged in advanced development planning.

6. 2010 NDIA Top SW Issues • Rapid acquisition: The increasing demands of the war fighter are requiring development of software and systems more rapidly than acquisition models and development processes allow. • Issues: responsiveness; lack multi-disciplinary approach; process mismatch; prioritization; adaptation; fielding prototypes; sustainability trade-offs. • Start programs right: Inadequate/insufficient program planning and start-up has a negative impact on the program’s SW success. • Issues: success-oriented plans; rework from late consideration of SW impacts; inconsistent use and maintenance of SDP; ineffective org structures, IMP/IMS; changing baselines; inadequate expertise; lack SW engagement in technical trades and risks

7. 2010 NDIA Top SW Issues • Execute programs right: Effective engineering practices are not well-defined or consistently implemented for key software processes with emerging mission needs and supply chain dependencies. • Issues: SoS, SW/SE I&T, system assurance, COTS/NDI integration; methodologies, scalability to complex systems. • Sustainment: There is insufficient attention given to the overall software life cycle activities including sustainment and changing threats. • Issues: sustainment resources are unable to keep up with changing mission scenarios; sustainment is not always considered adequately in system life cycle planning, engineering, and budgets; trades for sustainability may be negatively impacted by competitive prototyping.

8. NDIA Top SE Issues – 2010 Preliminary Results - NDIA Top SE Issues Workshop March 17-18, 2010

9. 2010 NDIA Top SE Issues • Rapid Fielding: • Increasingly urgent demands of the warfighter require effective capabilities be fielded more rapidly than the conventional acquisition processes and development methodologies allow. • Issues: quick response capability encumbered by tradition acquisition processes; SE process rigor may vary by acceptable risk; trades should consider impacts across entire SE life cycle; critical success factors for rapid response are not well documented or used in decision-making Recommendation: • NDIA Systems Engineering Division to convene a joint government/industry study, with cross-functional representation including T&E, to develop risk-driven guidance on tailoring SE processes and activities and DoD acquisition requirements from the Defense Acquisition System to achieve rapid acquisition and deployment.

10. 2010 NDIA Top SE Issues • Human Capital: • The quantity and quality of Systems Engineering expertise is insufficient to meet the demands of the government and defense industry. • Issues: SE skills, learning, and development; accelerating growth of quality SEs; interfaces between SE and SW. Recommendations: • Develop systems engineering expertise through role definition, selection, training, career incentives, and broadening “systems thinking” into other disciplines. • Establish systems engineering job codes or other indicators within government service to emphasize systems engineering in the recruiting of new hires.

11. 2010 NDIA Top SE Issues Issue 2, Human Capital: Recommendations (con’t.) • OSD should establish policy/guidance for services to include systems engineering expertise in RFP/proposal efforts and program start-up teams with reach-back capability. Consideration should be given to integrating this initiative with the OSD transition team concept. Discrete budgets should be allowed for the SE effort and fenced from cuts without a thorough review of the implications. • NDIA SED Education and Training Committee should investigate and recommend specific methods and techniques to accelerate the training and development of Systems Engineers to include purposeful OJT rotation programs to compliment course work. • Provide contract incentives to increase the development of systems engineers on programs.

12. 2010 NDIA Top SE Issues • Effective SE: Systems engineering practices known to be effective are not consistently applied or properly resourced to enable early system definition • Issues: SE best practices not consistently followed; competitive pricing can lead to tradeoffs in SE process fundamentals; SE value not consistently understood; dependencies on SE resources and experience Recommendations: • Identify key SE information that decision makers need and develop consistent methods for presenting that information. • Establish a working group to determine why proven tools and techniques are not consistently applied. • To create a receptive audience to gain full value from these recommendations to be effective, implementation of Issue 2 recommendations are encouraged.

13. 2010 NDIA Top SE Issues • Informed decision-making: • Decision makers do not have the right information at the right time to support informed and proactive decision making that ensures effective and efficient program planning, management and execution. • Issues: effective use of leading indicators to manage program success; lack a quantitative basis for decision-making; address systemic program technical issues. Recommendations: • NDIA will create a roadmap that programs can use to facilitate the definition and implementation of a technical measurement process to support engineering management decisions. The roadmap should be coordinated with INCOSE Measurement Working Group and PSM.

14. 2010 NDIA Top SE Issues Issue 4, Informed Decision Making, Recommendations (con’t.) • NDIA should create a research agenda to investigate causes and effects for program systems engineering success and characterization of SE methods, practices, and solution approaches. Investigate the identification of context factors and how they dynamically interact. Include NAVAIR. • NDIA should work with other parties to explore & potentially create a data collection and analysis capability in order to develop technical management benchmarks. Need to create the value proposition that justifies the considerable effort likely required (e.g., OSD/Systems Engineering root cause analysis data base, USC as part of the Systems Engineering Research Center, SEI - Software Engineering Institute, PSM - Practical Software and Systems Measurement).

15. 2010 NDIA Top SE Issues • SE technical integrity and rework: • Lack of technical authority can impact the integrity of developed systems and result in cost/schedule/system performance impacts as the technical solution is iterated and reworked in later stages of the development. • Issues: lack empowerment for proper SE; critical issues and tradeoffs not adequately addressed; excessive rework from not resolving issues early; independent technical authority to escalate SE technical issues. Recommendations • Add a requirement to the SEP guidance that the SEP must explicitly identify the technical authority (TA) for the program: • the TA role, authority, responsibility, and the performance reporting chain • TA conduct and chairing of technical reviews • the mechanism for elevating technical issues beyond the program office for resolution.

16. 2010 NDIA Top SE Issues • Issue 5, SE technical integrity and rework, Recommendations (con’t.) • There is a need to establish event-based technical reviews with enforced entry criteria and go deeper into technical details. • The government PMO (and contractor’s programs) must staff these reviews with experienced engineers who can operate somewhat independently of the Program Office and ensure that appropriate technical authority is being applied to the program. The matrix organization (people not directly doing the work) must support peer reviews. • NDIA review and update guidance, as needed, for entry/exit criteria and application for consistent rigor in technical reviews

17. 2010 NDIA Top SE Issues • Issue 5, SE technical integrity and rework, Recommendations (con’t.) • Create a career path for senior technical people with broad expertise and wide experience who can see strategic viewpoints need a career path that leads to seniority and authority equivalent to the Program Manager and PEO. • In practice these engineers are by definition (rules) always subordinated to the Program Manager; but there needs to be a technical authority chain of command to balance program authority, specifically the Chief Systems Engineer on the program. The Chief Engineer is presumed to possess the adequate skill set, the experience of a systems thinker / systems integrator, and is recognized as the technical authority for the program. • NDIA SE Division discuss with NDIA PM Systems Committee and OSD to determine if there are actions that can be taken through NDIA to support this.