Using databases to test methods for decision under uncertainty

1. Using databases to test methods for decision under uncertainty R. T. Haftka, R. I. Rosca, E. Nikolaidis September 2004

2. Introduction Decision under uncertainty Uncertain event Information Outcome Decision Consequence

3. Need unbiased and efficient testing approach for methods for decision under uncertainty • Decisions can be sensitive to modeling errors • Look good on paper but disappoint in practice • Tests can reveal weaknesses in methods that are difficult to identify by examining their theoretical foundations

4. Challenge • Large number of tests to obtain statistically meaningful results expensive to test methods • Proposed testing approach • Database with real life data • Invent decision scenario in which data represent outcomes of uncertain events/decisions • Use data to make decisions and evaluate payoffs

5. Database from student competition Competitor Rosca

6. Outline • Testing approach • Demonstration: Comparison of probability and possibility • Results and lessons learnt • Conclusion

7. 1. Testing approach A Step A: Find database Evaluate payoff of decision using entire database Step B: Create decision scenario Step C: Select fitting* (learning) subset No Completed all replications? Fitting subset No: Select another fitting subset Yes Step D: Estimate probability of winning (or expected utility), draw conclusions Model uncertainties, make decision A * Fitting subset contains incomplete information available to decision maker

8. A. Select database Uncertain variables: Maximum built heights of Rosca and Competitor

9. B. Create decision scenarioVariables in database represent uncertainties Consequences Competitor's tower collapses Rosca wins Rosca builds stable tower with height, nguar nguar, Rosca losses Decision: Rosca chooses guarantee height, nguar Competitor builds stable tower with height nguar + handicap+1 Rosca losses Rosca's tower collapses Rorca must select guaranteed height and get a handicap in return. She wins if she builds the tower she promised and is at least as high as the competitor’s minus the handicap.

10. Testing approach: Steps C and D A Step A: Find database Find if Rosca wins or losses all combinations of heights in database Step B: Create decision scenario No Step C: Select fitting subset Completed all replications? No: Select another fitting subset Yes Step D: Estimate probability of winning, draw conclusions Model uncertainties, make decision A

11. Abundant databases -- different decision problems • Student academic records • Student design projects in undergraduate class on mechanisms

12. Probabilistic formulation Select nguar To max Gamma and normal PDF for max built height Possibilistic formulation Select nguar To max Triangular possibility distribution 2. Demonstration: Comparison of probability and possibility Probability maximizes right objective function: probability of winning. Fitting error in approximating discrete PDF of max built height with a standard PDF.

13. Evaluation of consequences of a decision Database: 50 max built heights for Rosca, 90 heights for competitor For each decision, determine consequences for 5090=4,500 competitions Max possible number of decisions = Number of fitting datasets =

14. 3. Results and lessons learnt • Rosca has all data in database

15. Fitting Error

16. Observations • Little difference in performance of probability and possibility • Fitting error offsets natural advantage of probability

17. Effect of inflation of distributions • Epistemic uncertainty, inflate distribution

18. Observations • Inflation decreases likelihood of success • Opposite effect on optimum decision of two methods. • Increase uncertainty in Competitor performance: probabilistic optimum decreases, possibilistic optimum increases • Difference due to non additivity of possibility

19. Decision scenario accentuating difference in optimum decisions of two methods

20. Observations • Optimality condition: • Probability; equal derivatives of probability of Rosca’s success and Competitor’s failure. Focuses on hazard easier to control (Rosca fails to deliver guaranteed height). • Possibility; equal possibilities of Rosca’s success and Competitor’s failure. Non additivity of possibility responsible for difference. • Possibility yields counterintuitive results when uncertainty in Competitor’s performance is very high. Optimum dominated by largest uncertainty (Competitor’s performance).

21. Results and lessons learntRosca has 5 data points Rosca still knows that normal and Gamma distributions fit all data well.

22. Observations • Methods have practically same performance • Probability: type of probability distribution that fits all data is known • Possibility: decision maker cannot directly use information about type of probability distribution • Hybrid method (uncertainty in maximum built heights modeled using probability distributions; uncertainty in distribution parameters modeled using possibility distributions) could do better

23. 4. Conclusion • Efficient testing approach for methods for decision under uncertainty • Uses real life data • Allows testing methods on many decisions and evaluate consequences for many outcomes of uncertain events • Abundant easily accessible databases from different fields • Can learn useful lessons from tests • Inflation of probability distributions counterproductive. Opposite effect on probabilistic and possibilistic optimum decision. • Identified weakness in possibilistic approach by accentuating difference between probability and possibility. Probability focuses on hazard easier to control; possibility does not. Cause: non additivity of possibility. • Small modeling errors can offset natural advantage of probability that it maximizes right objective function.