Ensuring System Reliability through Rigorous Testing Methods

1. EMIS 7307 T&E Part 3 A few DT&E Areas of focus • Life testing. • To make sure that the system will not fail prematurely due to fatigue, aging, long term environmental exposure etc. • Time consuming and expensive. • Consider accelerated life testing.

2. EMIS 7307 T&E Part 3 A few DT&E Areas of focus • Design evaluation/verification. • See if critical characteristics are achievable • Provide data making the hardware or software more rugged in order to meet specs. • Eliminate risks. • Assist design evolution. • Ensure component and system meet spec.

3. EMIS 7307 T&E Part 3 A few DT&E Areas of focus • Design limit testing. • Based on mission profile. • Operated at the limits. • May be destructive. • Interpolation and extrapolation may be required.

4. EMIS 7307 T&E Part 3 A few DT&E Areas of focus • Reliability Development testing. • Also called reliability growth testing, test- analyze-fix-test (TAFT), of test analyze and fix (TAAF). • Test under mission profile environments. • Things fail. • Determine and make fix. • Commence testing again. • Data allows updating reliability model for current predictions.

5. EMIS 7307 T&E Part 3 Built-In Test • Difficult part of design • Difficult part of I&T • Difficult part of T&E • Issues include: • Level of fault isolation • Level of automatic fault detection • Induce failures for the T&E?

6. EMIS 7307 T&E Part 3 OT&E • Contractor perspective. • Usually involvement is limited to being the depot - a common approach. • If the program/system has contractor logistics support (CLS) then their performance is tested here too.

7. EMIS 7307 T&E Part 3 OT&E • OT&E government program office perspective. • ‘Make or break’ time. • This is the program office’s final exam! • Feels like you’re sending your kid to the army! • Testing is done at arms length from the program office. • No program office involvement except to pay for it!

8. EMIS 7307 T&E Part 3 OT&E • OT&E government operational tester perspective. • An agent of the highest authority. • One phrase in the final report and the system does not go to FRP. • Not operationally suitable, not operationally effective.

9. EMIS 7307 T&E Part 3 OT&E Supports Reviews and MSs Too

10. EMIS 7307 T&E Part 3 OT&E • Critical Operational Issues (COI) • A key operational effectiveness or operational suitability issue that must be examined in operational test and evaluation to determine the system’s capability to perform its mission. A critical operational issue is normally phrased as a question to be answered in evaluating a system’s operational effectiveness and/or operational suitability. • Identifying COIs • Ask: “How well does the system perform a particular aspect of its mission?” “Can the system be supported logistically in the field?” Other issues arise from questions asked about system performance or how it will affect other systems with which it must operate. Critical issues provide focus and direction for the operational test. Identifying the issues is analogous to the first step in the system engineering process — that is — defining the problem. When critical operational issues are properly addressed, deficiencies in the system can be uncovered.

11. EMIS 7307 T&E Part 3 OT&E • COIs are the basis for sub-objectives known as Measures of Effectiveness (MOEs) which in turn yield actual test measurements called Measures of Performance (MOPs).

12. EMIS 7307 T&E Part 3 OT&E

13. EMIS 7307 T&E Part 3 OT&E • Test Realism • Key issue, if it isn’t realistic, it’s not good OT&E. • Realistic production representative version • Realistic environment. • Realistic tactics and threats. • Realistic logistics. • Realistic training. • Realistic operators - not the ‘golden crew’.

14. EMIS 7307 T&E Part 3 Live Fire Testing (LFT) • A lot like OT&E. • Mandated by congress. • Prior to full rate production. • Appropriate threat weapons used against our. systems, configured for combat. • Or: • Our munitions are fired against a threat target.

15. EMIS 7307 T&E Part 3 Before IOT&E, data are gleaned wherever possible.

16. EMIS 7307 T&E Part 3 Evaluation • Remember the definitions of test, evaluation! • Establishing and maintaining a clear audit trail is essential from: • System requirements through critical issues. • Evaluation criteria. • Test objectives. • Measures of effectiveness. • Data are used from all sources, not just tests • Design reviews, inspection, simulations, etc.

17. EMIS 7307 T&E Part 3 Evaluation • Issues: • Questions that require answers during the acquisition process to support. • Development of acquisition strategy. • Refining performance requirements and designs. • MS decisions. • Criteria: • Standards of assessment for: • Technical and operational effectiveness/suitability characteristics. • Resolution of operational issues.

18. EMIS 7307 T&E Part 3 Evaluation • Key Performance Parameters (KPPs) from the ORD yield: • Critical issues. • Must be answered before the systems overall worth can be estimated or evaluated. • Of primary importance to the milestone decision authority. • Evaluation issues. • Technical implies DT, if possible. • Operational implies analysis, modeling, simulation. inspection, demo or test.

19. EMIS 7307 T&E Part 3 Evaluation • Test issues. • Issues that cannot or should not be addressed by any other method. • Criteria – Required level of technical performance and operational effectiveness, suitability or supportability. • Objective: Goal, a preferred level of capability. Not addressed by OT&E. • Threshold: Minimum acceptable level of acceptability. • System is not worth having below this level. • One function of T&E is to verify meeting the threshold.

20. EMIS 7307 T&E Part 3 Generic Evaluation Process

21. EMIS 7307 T&E Part 3 Is the real thing always better?

22. EMIS 7307 T&E Part 3 How to decide?

23. EMIS 7307 T&E Part 3 Modeling and Simulation Support Testing • Help discover sensitive areas of special interest. • Can go where real testing cannot. • Can be updated to improve validity based. on selected test data. • Can be used for design, test and operations.

24. EMIS 7307 T&E Part 3 • The final consists of two sections. • An in-class section (25 %) • 25 questions similar to midterms • Mostly concerning topics since the 2nd midterm • A four question essay section which you prepare and bring to the final (75 %) • Use MSWord and size 12 or 14 Times New Roman font. Both substance and grammar will be graded. Each person’s offering must be unique. • Number of words is approximate.

25. EMIS 7307 T&E Part 3 Essay portion of final exam (1)What is the role of simulation and modeling in the I&T process? (250 words) (2)What is the role of simulation and modeling in the T&E process? (250 words)

26. EMIS 7307 T&E Part 3 Essay portion of final exam (3)Pretend I’m your boss. Convince me to change our company’s approach to I&T. Tell me what’s wrong with our I&T. Provide pros and cons of how you would revise it, and then explain why the pros outweigh the cons. Show how this revised approach will improve our overall systems engineering flow and ultimately provide a better product. Include suggested organizational changes that will be required to implement this change. (250 words, plus a table and /or chart)

27. EMIS 7307 T&E Part 3 Essay portion of final exam (4)Discuss the pitfalls, dangers and risks of the “we’ll make up lost time” and “we’ll fix the problems during I&T” mentality. (250 words) • The remaining portion of the final will be administered in the same fashion as the midterms. After completing the remainder of the final, attach your essays to it for one combined submittal.