Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers

1. National Aeronautics and Space Administration Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C. Ariel Pinto and Resit Unal Old Dominion University James T. Luxhøj Rutgers University Hampton Roads Area INCOSE Conference on Decision Analysis and Its Applications to Systems Engineering, Newport News, VA (November 17-18, 2009) www.nasa.gov

2. Outline • Background/Definitions • Data Collection Process • Results • Concluding Remarks Jones,, et al. (2009)

3. Problem Definition • Aerospace technology managers need: • To predict future technologies • To assess progress toward R&D goals • Aerospace technology portfolio decisions are difficult because: • Very little time to acquire background data • Uncertainty factors (e.g., politics, global economy, environment, funding) • Prescriptive decision analysis methods • Have been used for financial portfolio assessment • Value for policy related decisions has been questioned Jones, et al (2009)

4. Ralph Keeney’s Suggestions for Making Better Decision Makers* • Develop concepts, tools and procedures to help decision makers • “My experience is that many people, including well-educated people, have a very difficult time in structuring their decisions. They can get mixed up about the difference between fundamental concepts such as alternatives and objectives.” • Use real decisions, not just laboratory problems in decision research • “We have learned a great deal from all the laboratory settings where decision experiments have been conducted. There have also been some research studies of real decision problems. I feel there is much more to be gained by having more of this type of research.” • Teach people what they can and will learn and use • “…hundreds and thousands of people have had at least a course that included a substantial part on decision analysis and very few have probably ever conducted a formal decision analysis. Once we find out what people can and will learn and use, that should constitute the basis for much of our teaching of decision analysis.” *Source: Ralph L. Keeney, “Making Better Decision Makers”, Decision Analysis, 1:4 (2004) Jones,, et al. (2009)

5. Decision Analysis Usage in Aerospace Portfolio Development • Aerospace managers have investigated the use of decision analysis methods for portfolio investment decisions: • Commercial Aviation Safety Team (CAST) • NASA Aviation Safety Program • Future Aviation Safety Team (FAST) • These technology assessments involved: • Resource commitments (e.g., employee time, travel money, software acquisition, training) • Assumption that decision analysis methods would improve aerospace technology assessment process Jones,, et al. (2009)

6. Technology Assessment “ A process for measuring the impact of established or new technologies” *Hans Mohr, “Technology Assessment in Theory and Practice”, Society for Philosophy and Technology, 4:4 (Summer 1999) Jones, et al (2009)

7. Aerospace Technology Assessment Three different processes for examining impact of a set of technologies • Technology assessment • Technology forecasting • Technology foresight Aerospace Technology Assessment Jones, et al (2009)

8. 1. Identify Context & Understand Objectives 2. Identify Alternatives 3. Decompose & Model the Problem a. Model of Problem Structure b. Model of Uncertainty c. Model of Preferences Decision Analysis Methods Aerospace Technology Assessment 5. Sensitivity Analysis 4. Choose the Best Alternative 6. Implement the Chosen Alternative Conceptual Model * *Adapted from Robert T. Clemen, Making Hard Decisions, 2nd Edition (1995) Jones, et al (2009)

9. What is known: Decision analysis methods in financial portfolio assessment Decision experiments in laboratory settings Technology assessment in medical R&D What is unknown: Decision analysis methods for policy related decisions Decision analysis methods in real decision problems Technology assessment in aerospace Purpose of Study Jones, et al (2009)

10. Study Overview • Population was aerospace researchers with experience in one or more of the following: • Aerospace program/project management • Aerospace technology assessment • Aerospace technology selection • Aerospace R&D portfolio development • Methods that were investigated in the study • Decision trees • Influence diagrams • Criteria aggregation methods • Explicit tradeoff approaches • Participants were questioned about their usage of these methods for aerospace technology assessment Jones, et al (2009)

11. Decision Trees = decision node = chance node = consequence node Jones, et al (2009)

12. Influence Diagrams chance node consequence node decision node Jones, et al (2009)

13. Criteria Aggregation Methods • Methods in which two sets of aggregated indices are developed and used to evaluate the alternatives in the decision problem • Methods in the category include: • Analytical Hierarchy Process (AHP) • Weighted Sum Model (WSM) Example of Simple Weighted Sum Model * *E. Triantaphyllou, Multi-Criteria Decision Making Methods: A Comparative Study(1995) Jones, et al (2009)

14. Explicit Tradeoff Approaches • Decision analysis methods that are based on value functions • Methods in this category include: • Multi-Attribute Utility Theory (MAUT) • Simplified Multi-Attribute Rating Approach (SMART) Jones, et al (2009)

15. Excluded Decision Analysis Methods • Avoided decision analysis methods that are not popular in U.S. • Real world applications are complex with large amounts of uncertainty • Specific decision analysis methods that were excluded from study: • Outranking methods (e.g., ELECTRE, TOPSIS) • Optimization methods • Analytic network process (ANP) Jones, et al (2009)

16. Develop List of Pilot Participants Conduct Pilot Survey Review & Analyze Pilot Results Modify Survey Instrument Develop Web-Based Survey Instrument Data Collection and Analysis Process Refine List of Candidate Survey Participants Conduct Survey Analyze Results Legend • Survey Development • Data Collection • Data Analysis Jones, et al (2009)

17. Web-Based Instrument Development • Several web-based services examined • Questions developed based on several sources: • Short surveys at professional meetings • Validated research in decision analysis literature • Identities of survey participants remained anonymous Jones, et al. (2009)

18. Pilot Survey • Conducted with subset of population (10 persons) • Think aloud cognitive interviewing techniques used • Solicitation of all thoughts and comments • Manual recording of information during completion of online survey • De-identification of subjects in final documentation Jones, et al. (2009)

19. Survey Instrument Modification • Questions were modified, added or eliminated from the survey based on input from: • Pilot survey comments • Data analysis of pilot survey data • Additional comments from other reviews (e.g., ODU IRB) • Number of survey questions reduced from 70 to 65 Jones, et al. (2009)

20. Data Collection Overview • Approval to conduct survey was obtained from NASA Langley and ODU Institutional Review Boards (IRB’s) • E-mail invitation was sent to 260 persons • 154 total visits to survey website • 16 partial responses • 99 completes surveys • Out of the 99 completed surveys • 76% male, 24% female • Highest degree level was 60% Masters, 21% Bachelors, 18% Doctorate, 1% Associates • 72% employed as government civil servants Jones, et al. (2009)

21. Job Functions Jones, et al. (2009)

22. Aerospace Experience Jones, et al. (2009)

23. Decision Trees Jones, et al. (2009)

24. Influence Diagrams Jones, et al. (2009)

25. Criteria Aggregation Methods Jones, et al. (2009)

26. Explicit Tradeoff Approaches Jones, et al. (2009)

27. Usage of Decision Analysis Methods Outside of a Classroom Environment Jones, et al. (2009)

28. Categories of Non-ATA Usage of DA Outside of a Classroom Environment Jones, et al (2009)

29. Additional Questions Jones, et al. (2009)

30. Additional Questions (cont’d) Jones, et al. (2009)

31. Additional Questions (cont’d) Jones, et al. (2009)

32. Additional Questions (cont’d) Jones, et al. (2009)

33. Additional Questions (cont’d) Jones, et al. (2009)

34. Likelihood of Future Usage of DA Methods Jones, et al. (2009)

35. Concluding Remarks • This is a subset of the total data • There are many additional questions in the study • More formal analysis of the data was conducted using structural equation modeling techniques to test a set of hypotheses • Survey participants believed that the successful use of decision analysis methods depends on: • Selection criteria in the decision model • Experience of the person that implements the method • Reliability of the input data • Training/education does not guarantee future use of a decision analysis method Jones, et al. (2009)

36. Questions?

37. SignOffPage Your Title Here