Quantitative Methods in Defense and National Security (2010)

1. Quantitative Methods in Defense and National Security (2010) Update from the Panel on Industrial Methods for the Effective Test and Development of Defense Systems Mike Cohen, study director 5/25/2010

2. Previous Related Work by CNSTAT Let me start with some recommendations from previous efforts that are relevant to the current study: 1998: Statistics, Testing, and Defense Acquisition – New Approaches and Methodological Improvements Conclusion 2.2: The operational test and evaluation requirement, stated in law, that the Director, Operational Test and Evaluation certify that a system is operationally effective and suitable often cannot be supported solely by the use of standard statistical measures of confidence for complex defense systems with reasonable amounts of testing resources.

3. 1998 Study (continued) Conclusion 3.1: Major advances can be realized by applying selected industrial principles and practices in restructuring the paradigm for operational testing and the associated information gathering and evaluation process in the development of military systems.

4. 1998 Study (continued) Recommendation 2.1: DoD and the military services should provide a role for operational test personnel in the process of establishing verifiable, quantifiable, and meaningful operational requirements. Although the military operators have the final responsibility for establishing operational requirements, the Operational Requirements Document would benefit from consultation with and input from test personnel, the Director, Operational Test and Evaluation, and the operational test agency in the originating service. This consultation will ensure that requirements are stated in ways that promote their assessment.

5. 1998 Study (continued) Recommendation 3.1: … The primary mandate of the Director, Operational Test and Evaluation should be to integrate operational testing into the overall system development process to provide as much information as possible as soon as possible on operational effectiveness and suitability. In this way, improvements to the system and decisions about continuing system development or passing to full-rate production can be made in a timely and cost-efficient manner.

6. 1998 Study (continued) Recommendation 3.3: The DoD and the military services, using common financial resources, should develop a centralized testing and operational evaluation data archive for use in test design and test evaluation. Recommendation 3.4: All services should explore the adoption of the use of small-scale testing similar to the Army concept of force development test and experimentation.

7. 1998 Study (continued) Recommendation 7.1: DoD and the military services should give increased attention to their reliability, availability, and maintainability data collection and analysis procedures because deficiencies continue to be responsible for many of the current field problems and concerns about military readiness. Recommendation 8.4: Service test agencies should be required to collect data on system failures in the field that are attributable to software. These should be recorded and maintained in a central database that is accessible, easy to use, and makes use of common terminology across systems and services. This database should be used to improve testing practices and to improve fielded systems.

8. Innovations in Soft. Engineering for Defense Systems (2003) Recommendation 1: Given the current lack of implementation of state-of-the-art methods in software engineering in the service test agencies, initial steps should be taken to develop access to --- either in-house or in a closely affiliated relationship --- state-of-the-art software engineering expertise in the operational or developmental service test agencies. Recommendation 2: Each service’s operational or developmental test agency should routinely collect and archive data on software performance, including test performance data and data on field performance. The data should include fault types, fault times, and frequencies, turnaround rate, use scenarios, and root cause analysis. Also, software acquisition contracts should include requirements to collect such data.

9. Innovations in Soft. Engineering for Defense Systems (cont) Recommendation 6: DoD needs to examine the advantages and disadvantages of the use of methods for obligating software developers under contract to DoD to use state-of-the-art methods for requirements analysis and software testing, in particular, and software engineering and development more generally.

10. Testing of Defense Systems in an Evolut. Acquis. Environment (2006) Conclusion 1: In evol. acquisition, the entire spectrum of testing activities should be viewed as a continuous process of gathering, analyzing, and combining information in order to make effective decisions. The primary goal of test programs should be to experiment, learn about the strengths and weaknesses of newly added capabilities or (sub)systems, and use the results to improve overall system performance. Furthermore, data from previous stages of development, including field data, should be used in design, development and testing at future stages. Operational testing (testing for verification) of systems still has an important role to play in the evolutionary environment, although it may not be realistic to carry out operational testing, comprehensively at each stage of the development process.

11. Testing of Defense Systems in an Evolut. Acquis. Environment (cont) Conclusion 2: Testing early in the development stage should emphasize the detection of design inadequacies and failure modes. This will require testing in more extreme conditions than those typically required by either developmental or operational testing, such as highly accelerated stress environments.

12. Testing of Defense Systems in an Evolut. Acquis. Environment (2006) Conclusion 3: To have a reasonable likelihood of fully implementing the paradigm of testing to learn about and to improve systems prior to production and deployment, the roles of DoD and congressional oversight in the incentive system in defense acquisition and testing must be modified. In particular, incentives need to be put in place to support the process of learning and discovery of design inadequacies and failure modes early and throughout system development.

13. Testing of Defense Systems in an Evolut. Acquis. Environment (2006) Recommendation 3: The undersecretary of defense (acquisition, technology, logistics) should develop and implement policies, procedures, and rules that require contractors to share all relevant data on system performance and the results of modeling and simulation developed under government contracts, including information on their validity, to assist in system evaluation and development.

14. Testing of Defense Systems in an Evolut. Acquis. Environment (2006) Recommendation 4: The undersecretary of defense (AT&L) should require that all technologies to be included in a formal acquisition program have demonstrated sufficient technological maturity before the acquisition program is approved or before the technology is inserted in a later stage of development. The decision about the sufficiency of technological maturity should be based on an independent assessment from the director of defense research and engineering or special reviews by the director of operational test and evaluation (or other designated individuals) of the technological maturity assessments made during the analysis of alternatives and during developmental test.

15. Testing of Defense Systems in an Evolut. Acquis. Environment (2006) Conclusion 5: The DoD testing community should investigate alternative strategies for testing complex defense systems to gain, early in the development process, an understanding of their potential operational failure modes, limitations, and level of performance.

16. Testing of Defense Systems in an Evolut. Acquis. Environment (2006) Recommendation 6: (a) To support the implementation of evolutionary acquisition, DoD should acquire, either through hiring in-house or through consulting or contractual or contractual agreements, greater access to expertise in the following areas: (1) combining information from various sources for efficient multistage design, statistical modeling, and analysis; (2) software engineering; and (3) physics-based and operational-level modeling and simulation …

17. Problems • we have been operating at a relatively high level – maybe we can try to change what is done on a day-to-day basis • we have been too statistically-oriented --- DoD system development needs to utilize better engineering practices, and • no matter what memoranda and guidances are written, you need to help people do their jobs better, since the guidances often don’t have much direct impact on practice.

18. Industrial Methods for the Eff. Test and Devlop. of Def. Systms (2010) ROSTER Vijay Nair, chair (Univ of Michigan) Pete Adolph (consultant) Peter Cherry (SAIC) John Christie (LMI) Tom Christie (consultant) Blanton Godfrey (NC State Univ) Raj Kawlra (Chrysler) John Rolph (USC) Elaine Weyuker (AT&T) Marion Williams (IDA) Alyson Wilson (Iowa State Univ)

19. Industrial Methods for the Eff. Test and Develop. of Defense Systems Charge: To plan and conduct a workshop that will explore ways in which developmental and operational testing, modeling and simulation, and related techniques can improve the development and performance of defense systems. Workshop --- four key talks by software and hardware engineers from industry.

20. Three Key Questions - Question 1: Finding Failure Modes Earlier 1. The earlier that failure modes and design flaws are identified during the development of defense systems, the less expensive are the design changes that are needed for correction. The workshop will explore what techniques are applied in industrial settings to identify failure modes earlier in system development and whether such techniques are system dependent or whether some generally applicable principles and practices can be discovered.

21. QUESTION 1 (cont) We will want to understand the extent to which it is important to utilize operational realism in testing schemes ---what realism needs to be present, what can be safely simulated, and what can be safely ignored? Also, what is meant by the envelope of operational performance for a system and how far beyond that envelope should one test to discover design flaws and system limitations? Finally, how are accelerated testing ideas utilized in industry; what are the advantages and disadvantages (besides that accelerated scenarios are extrapolations from real use), and what are the pitfalls and ways around them?

22. Question 2: Assessment of Technological Maturity 2. The inclusion of hardware and software components that are not technologically mature is often the cause of delays in system development, cost increases, and reduced system performance when fielded. …Therefore, one key issue is how to set requirements and specifications for a new technology. A second key issue is how much of testing should be of the components as isolated systems and how much should be devoted to tests of the components within the functioning of the parent system? The workshop will explore the techniques that are applied in industrial settings to evaluate components efficiently to determine whether they are or are not sufficiently mature for use. This should include a discussion of how the assessment of maturity of software components is different than that of hardware components.

23. Question 3: Use of Information from Disparate Sources 3. Field performance data can be extremely useful for identifying design deficiencies for systems developed in stages. However, field use is not always conducive to data collection, resulting in various forms of incomplete data. How should feedback mechanisms supported by field performance data operate to improve modeling and simulation and developmental and operational test and evaluation practices? Further, along with field performance data, system performance can be gauged using results from developmental testing, operational testing, modeling and simulation, and the results of these various sources for earlier stages of development and for closely related systems or systems with identical components. However, these disparate sources of information are from the operation of a system in very different contexts and may also involve appreciably different systems, and it is therefore a challenge to develop statistical models for combining these data. … How can these disparate sources of information be used to help improve operational test design? What statistical models are useful for operational evaluation?

24. Topics We Hope to Address:1. Setting Requirements Setting realistic, useful requirements a. Use of high-level models to assess the feasibility of a proposed system b. Input from testers to assess testability c. Input from ultimate users for early design changes d. What if requirements ‘need’ to be changed mid-development? Need to be able to say yes or no from an engineering perspective

25. 2. Assessment of Technological Maturity a. Reliability vs. Effectiveness in assessing technological maturity– need for greater focus on reliability b. Greater use of alternative acquisition processes (ACTD) c. There are already procedures on the books --- are they followed? TRL level assessment is defined – unfortunately they are not quantitatively defined and so are easier to be fuzzy about d. Expertise greatly needed here --- is this idea likely to be operationalized soon or not?

26. 3. Finding Failure Modes and Design Flaws Earlier • Focusing on what the developers found in their private testing – continuity of learning --- and testing those components in those situations more to see if adjustments worked • Asking developers to use an agile-like development process --- asking to see the performance of components, and then subsystems, and then whole systems at various stages of the development process for independent testing.

27. 3. Finding Failure Modes and Design Flaws Earlier Hardware Questions (we may not be able to answer all): i. Testing at the ‘edge of the envelope’ --- how do we define that? Is that always system specific? ii. How much operational realism is needed to find failure modes of various types? When can we test components in isolation, when do we have to use typical users? iii. What is the proper role of accelerated testing? iv. What is the proper role of physics-based modeling and simulation for finding failure modes early in development? v. Support feedback loops for system improvement vi. Support feedback loops for test process Improvement

28. 3. Finding Failure Modes and Design Flaws Earlier Software Questions: i. what is the role of regression testing based on earlier stages of development? ii. how should we test more heavily in problematic areas of the software system? iii. we CANNOT rely on OT (or maybe even current DT) to exercise software with nearly enough replications, so we need some type of automated representation of the remainder of the system to do fully automated testing.

29. 3. Finding Failure Modes and Design Flaws Earlier Integration Questions: i. It may be hardest to find integration problems early since it requires more of the system to be developed. ii. It may require more creative representation of subsystems and associated expertise to determine where the integration issues are likely to reside and then test more there. (If you use a heavier payload than a component was previously required to accommodate, where might problems show up?

30. 4. Improving System Reliability a. Role of reliability growth models? b. Greater emphasis in DT – don’t pass into OT until DT isn’t finding any reliability problems? c. Accelerated testing? d. Component testing vs. Subsystem testing vs. System testing

31. 4. Improving System Reliability (cont) e. Some reliability problems will NOT show up in DT unless you know what realism needs to be represented. Testing needs to be looked at comprehensively and collaboratively --- DT and OT need to strategize f. FMECA analysis --- how to build this into the current process – wonderful graphics now at Aberdeen g. Retrieve defective parts and determine what went wrong h. Focus on life-cycle costs – more resources devoted to testing will easily pay for itself

32. 5. Process Changes a. Stability in the PM position --- need for greater accountability at the top i) Civilian with near equal authority in a position that continues throughout development, maybe chief engineer ii) Deputy PM gets promoted to PM b. Incongruent incentives --- PMs are not motivated to find every error One possibility --- Independent comprehensive assessment of the system status when the PM turns over. The PM is then judged more objectively about how the system progressed under them, especially suitability design flaws. c. Greater engineering expertise Those managing development should be experienced in various types of system development d. Ensure that current critical processes are followed --- e.g, TRL level should be 7, but maybe a 5 will still get a pass, etc.

33. 5. Process Changes (cont) e. Greater collaboration between contractors, testers, and the ultimate user. Place more constraints in contract to require sharing of test designs, test results, M&S development, M&S results, etc., etc. This really does not have a downside. It needs to be done. f. Everyone works from the same database, with everyone having easy access, so that requirements, specifications, and later assertions of reliability and effectiveness based on archived test results of various kinds can be understood and debated g. More realism about schedules and a view to do it right the first time --- no free passes to next stages of development without proof.

34. In conclusion Wish us luck --- we should be done in the fall. Thanks.