Risk and Uncertainty in Design Trade Studies

1. Risk and Uncertainty in Design Trade Studies Douglas Van Bossuyt PhD Qualifier June 11, 2009

2. My Background • OSU Triple Play • Research Interests • Collaborative design • Complex system design • Design for cultures • Psychology • Business management

3. Outline • Overview of Design Trade Studies • Methods, Tools, Groups Who Perform, Examples • Overview of Risk and Uncertainty • Risk quantification and mitigation tools • Uncertainty assessment methods • Uncertainty mitigation tools • Possible ways to incorporate risk and uncertainty into Trade Studies • Future areas of research • Expected contributions of overall research • Research plan

4. Introduction • Complex systems are here to stay • Every complex system design tries to maximize system utility • System utility metrics: ROI, system integrity, public perception of project, etc… • Subsystems optimized to achieve high overall system utility

5. Introduction (Con’t) Design parameters (mass, power, volume, cost, heat dissipation, etc.) used to define subsystem parameters that determine system-level utility Parameters are traded between subsystems to optimize design in Trade Studies Risk and uncertainty of systems is another important factor in complex system design

6. Introduction (Con’t) • Definitions • Risk: probability of event X impact of event • Sometimes more narrowly means probability of catastrophic event X impact of event • Uncertainty: caused by variability and doubt in the status of an output that is either predictable or unpredictable, or caused by an unknown process or device • Reliability: Ability of a device to perform as intended over a given period of time

7. Introduction (Con’t) • Definitions (Con’t) • Robustness: Ability of a device to continue to properly function under changes in input variables. • Design Margins: Quantify the influence of uncertainties in the design process. Often a high and low bounding around a central parameter quantification.

8. Motivation Methods developed from this research will improve system utility and integrity Improved utility and integrity uses resources more efficiently and produces more desirable results

9. Outline • Overview of Design Trade Studies • Methods, Tools, Groups Who Perform, Examples • Overview of Risk and Uncertainty • Risk quantification and mitigation tools • Uncertainty assessment methods • Uncertainty mitigation tools • Possible ways to incorporate risk and uncertainty into Trade Studies • Future areas of research • Expected contributions of overall research • Research plan

10. Design Trade Studies • Trade Studies in complex systems design • Both design and decision tool • Trade Studies attempt to find maximum system utility • System utility defined by many metrics: • Cost • Return on Investment • System Reliability

11. Trade Studies Methods • Multi-step process to perform a Trade Study • Understand system goals, objectives and constraints (Eg: Functional requirements) • Develop alternative conceptual design solutions • Evaluate alternatives based on system utility • Use mathematical models where appropriate to determine system utility • Rank alternatives according to their system utility • Remove less desirable alternatives • Either refine and continue to eliminate alternatives or select most desirable alternative

12. Trade Studies Methods (Con’t) Image Source: NASA Systems Engineering Handbook

13. Trade Studies Tools • Trade Studies search for max system utility • Many mathematical ways to find max utility • Modern software packages available to find optimum design points • ICEMaker: Used by many Collaborative Design Centers to find optimum designs • Advanced Trade Space Visualization (ATSV): Used to graphically view and explore optimum design points

14. Trade Studies Tools (Con’t) ATSV Screenshots Image Source: https://webhosting.its.psu.edu/atsv/webfiles/glyphscatter /WebStart_files/image003.jpg

15. Trade Studies Tools (Con’t) • Software (Con’t) • ModelCenter: Integrates capabilities of ATSV with ability to link together many different types of programs • Many other types of software available that help perform Trade Studies

16. Trade Studies Tools (Con’t) Image Source: Jensen, et. al.: ME 519 Class Project

17. Groups Who Perform Trade Studies • Many CDCs exist in government organizations, academia, and industry • Original is Team-X housed at NASA JPL • Helped NASA reduce time to finish Trade Studies from 3-9 months to 2-3 days • Reduced cost by a factor of five • Other NASA facilities with CDCs: Langley Research Center, Goddard, Johnson Space Center

18. Groups Who Perform Trade Studies (Con’t) Image Source: http://jplteamx.jpl.nasa.gov/images/teamx/team.jpg

19. Groups Who Perform Trade Studies (Con’t) European Space Agency uses CDCs and Trade Studies Boeing, Aerospace Corporation, TRW, and other aerospace companies use Trade Studies Several academic institutions also use trade studies

20. Examples of Trade Studies • Many examples in literature of Trade Studies • Most come from Team-X but some also from academic institutions • Very few from private industry • Due to proprietary information, etc • For those interested, long list of Trade Studies is available

21. Outline • Overview of Design Trade Studies • Methods, Tools, Groups Who Perform, Examples • Overview of Risk and Uncertainty • Risk quantification and mitigation tools • Uncertainty assessment methods • Uncertainty mitigation tools • Possible ways to incorporate risk and uncertainty into Trade Studies • Future areas of research • Expected contributions of overall research • Research plan

22. Risk and Uncertainty in Collaborative Design and Model Based Engineering Overview of methods to account for risk Overview of uncertainty and how to account for it in design process Note: many methods not reviewed here due to space and time constraints

23. Risk Methods and Tools • Many methods and tools available • Some used in practice, some only in academia • Practice • RBD, Databases, FMEA/FMECA , ETA, FTA, PRA, QRA • Theory • FFDM, FFIP, RED, HiPHOPS, RUBIC, FFA

24. Reliability Block Diagrams Used for understanding fault tolerance Energy, information, or material flow through block diagram Image source: http://www.itemsoft.com/rbd.shtml

25. Databases Contain failure and reliability data on systems, subsystems, components, and processes Proprietary and industry-specific High amount of front-end work to have worthwhile database Often used in oil, automotive, and aerospace industries

26. Failure Modes and Effects (Criticality) Analysis • FMEA used to examine: • Potential failures modes • Effects of failures • Severity of the effects • Potential causes of the failures • Probability or potential probability of failure • Current detection methods of failure • Detectability of failure • Recommendations to mitigate cause or effects of failure

27. FMEA/FMECA (Con’t) • FMEA also can be used to assign a Risk Priority Number • RPN = Severity x Occurrence x Detection • Severity of each failure is rated • Likelihood of each occurrence is rated • Likelihood of prior detection is rated • FMECA is an extension of FMEA. Adds criticality analysis to FMEA. • Mode Criticality = Expected Failures X Mode Ratio of Unreliability X Probability of Loss • Item Criticality = SUM of Mode CriticalitieS

28. FMEA/FMECA (Con’t) Image Source: http://www.weibull.com/basics/fmea_fig1.htm

29. Event Tree Analysis Image Source: http://www.event-tree.com/images/et_example.JPG ETA is visual representation of failure events and mitigating events in a system Used in safety system analysis Starting point is failure event Subsequent levels show additional failures and mitigations

30. Fault Tree Analysis Image Source: http://www.isograph-software.com/ftpoverdgc.htm FTA starts with failure at top-level and proceeds down to analyze all possible causes of failure Boolean operators and logic gates used

31. Probabilistic Risk Assessment PRA is used to quantify the risk of failure in a system Employs FTA, ETA, and other techniques as desired PRA quantifies risk by magnitude and likelihood of each possible failure PRA is essentially an umbrella for several other risk methods

32. Qualitative Risk Assessment • Used when quantitative assessment is not possible • Not enough time, money, expertise • Relies on expert opinions • Usually performed by interviewing key designers to determine their belief in the level of risk of a design

33. Function Failure Design Method FFDM used to investigate potential failure modes during conceptual design Uses failure databases to find failure rates of generic components Improves on FMEA and related techniques

34. FFDM (Con’t) Image Sources: Stone, Tumer, Van Wie: The Function-Failure Design Method

35. Function Failure Identification Propagation • FFIP estimates potential failures and their propagation paths through systems • Three components to FFIP: • Graphical system model • Behavioral simulation • Reasoning scheme called Function Failure Logic

36. FFIP (Con’t) Image Sources: Kurtoglu and Tumer:A Graph-Based Fault Identification and Propagation Framework for Functional Design of Complex Systems

37. FFIP (Con’t)

38. Risk in Early Design An extension of FFDM Quantifies risks identified in FFDM Automated process for combining historical risk data with new system architectures Uses fever charts to show risks Displays riskiest failure states

39. RED (Con’t) Image Source: Lough, Stone, Tumer: Implementation Procedures for the Risk in Early Design (RED) Method

40. Hierarchically Performed Hazard Origin and Propagation Studies HiPHOPS uses elements of FMEA, FTA, and others to assess risk in systems Model of system is annotated with formalized logical component failure descriptions and expected effects This method is too complex to ever gain widespread adoption

41. Risk and Uncertainty Based Integrated and Concurrent Design Methodology A continuous risk management tool Used to identify risk elements during conceptual design RUBIC continuously optimizes budgetary resources to mitigate risks Graphical tool helps find Pareto optimal sets of resource allocations

42. RUBIC (Con’t) Image Source: Mehr, Tumer: Risk-Based Decision-Making for Managing Resources During the Design of Complex Space Exploration Systems

43. Functional Fault Analysis FFA captures physical system architecture including connections of energy, material, and data flows in a functional model Model contains sensor information, failure modes of each component, propagation effects of failure modes, and propagation timing Approach requires high level of detail in system before it is useful

44. Other Risk Methods State Event Fault Tree Analysis Component Fault Tree Analysis Simulation-Based Probabilistic Risk Analysis Component Stress and Conceptual Strength Interference Theory Various Bayesian Network Analysis tools Many others

45. How Risk Methods Relate All try to identify and quantify risk All good for identifying riskiest points in designs In practice, lists of failures versus failure paths methods Most theoretical tools trying to find subsystem and component interaction risks

46. Outline • Overview of Design Trade Studies • Methods, Tools, Groups Who Perform, Examples • Overview of Risk and Uncertainty • Risk quantification and mitigation tools • Uncertainty assessment methods • Uncertainty mitigation tools • Possible ways to incorporate risk and uncertainty into Trade Studies • Future areas of research • Expected contributions of overall research • Research plan

47. Uncertainty Image Source: http://www.martin-koser.de/images/enjoy%20uncertainty.jpg Definitions of uncertainty Assessing System Uncertainty Mitigating Uncertainty

48. Uncertainty (Con’t) • Many different ways to define uncertainty and many different places for it to be found • Easiest to think of uncertainty as being made of many different types and falling into two categories • Categories: • Intrinsic: Caused by randomness in nature • Epistemic: Caused by lack of knowledge or data

49. Uncertainty (Con’t)

50. Assessing System Uncertainties • Several ways to assess uncertainties: • Probabilistic Methods • Bayesian Techniques • 1st, 2nd, 3rd level Bayesian Analysis • Bayesian Team Support • Stimulation Methods • Monte Carlo Methods