1 / 23

Using databases to test methods for decision under uncertainty

This paper explores the use of databases to test methods for decision making under uncertainty. It presents a testing approach using real-life data and demonstrates a comparison of probability and possibility. The results and lessons learned from the testing are discussed.

kspradlin
Download Presentation

Using databases to test methods for decision under uncertainty

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Using databases to test methods for decision under uncertainty R. T. Haftka, R. I. Rosca, E. Nikolaidis September 2004

  2. Introduction Decision under uncertainty Uncertain event Information Outcome Decision Consequence

  3. Need unbiased and efficient testing approach for methods for decision under uncertainty • Decisions can be sensitive to modeling errors • Look good on paper but disappoint in practice • Tests can reveal weaknesses in methods that are difficult to identify by examining their theoretical foundations

  4. Challenge • Large number of tests to obtain statistically meaningful results expensive to test methods • Proposed testing approach • Database with real life data • Invent decision scenario in which data represent outcomes of uncertain events/decisions • Use data to make decisions and evaluate payoffs

  5. Database from student competition Competitor Rosca

  6. Outline • Testing approach • Demonstration: Comparison of probability and possibility • Results and lessons learnt • Conclusion

  7. 1. Testing approach A Step A: Find database Evaluate payoff of decision using entire database Step B: Create decision scenario Step C: Select fitting* (learning) subset No Completed all replications? Fitting subset No: Select another fitting subset Yes Step D: Estimate probability of winning (or expected utility), draw conclusions Model uncertainties, make decision A * Fitting subset contains incomplete information available to decision maker

  8. A. Select database Uncertain variables: Maximum built heights of Rosca and Competitor

  9. B. Create decision scenarioVariables in database represent uncertainties Consequences Competitor's tower collapses Rosca wins Rosca builds stable tower with height, nguar nguar, Rosca losses Decision: Rosca chooses guarantee height, nguar Competitor builds stable tower with height nguar + handicap+1 Rosca losses Rosca's tower collapses Rorca must select guaranteed height and get a handicap in return. She wins if she builds the tower she promised and is at least as high as the competitor’s minus the handicap.

  10. Testing approach: Steps C and D A Step A: Find database Find if Rosca wins or losses all combinations of heights in database Step B: Create decision scenario No Step C: Select fitting subset Completed all replications? No: Select another fitting subset Yes Step D: Estimate probability of winning, draw conclusions Model uncertainties, make decision A

  11. Abundant databases -- different decision problems • Student academic records • Student design projects in undergraduate class on mechanisms

  12. Probabilistic formulation Select nguar To max Gamma and normal PDF for max built height Possibilistic formulation Select nguar To max Triangular possibility distribution 2. Demonstration: Comparison of probability and possibility Probability maximizes right objective function: probability of winning. Fitting error in approximating discrete PDF of max built height with a standard PDF.

  13. Evaluation of consequences of a decision Database: 50 max built heights for Rosca, 90 heights for competitor For each decision, determine consequences for 5090=4,500 competitions Max possible number of decisions = Number of fitting datasets =

  14. 3. Results and lessons learnt • Rosca has all data in database

  15. Fitting Error

  16. Observations • Little difference in performance of probability and possibility • Fitting error offsets natural advantage of probability

  17. Effect of inflation of distributions • Epistemic uncertainty, inflate distribution

  18. Observations • Inflation decreases likelihood of success • Opposite effect on optimum decision of two methods. • Increase uncertainty in Competitor performance: probabilistic optimum decreases, possibilistic optimum increases • Difference due to non additivity of possibility

  19. Decision scenario accentuating difference in optimum decisions of two methods

  20. Observations • Optimality condition: • Probability; equal derivatives of probability of Rosca’s success and Competitor’s failure. Focuses on hazard easier to control (Rosca fails to deliver guaranteed height). • Possibility; equal possibilities of Rosca’s success and Competitor’s failure. Non additivity of possibility responsible for difference. • Possibility yields counterintuitive results when uncertainty in Competitor’s performance is very high. Optimum dominated by largest uncertainty (Competitor’s performance).

  21. Results and lessons learntRosca has 5 data points Rosca still knows that normal and Gamma distributions fit all data well.

  22. Observations • Methods have practically same performance • Probability: type of probability distribution that fits all data is known • Possibility: decision maker cannot directly use information about type of probability distribution • Hybrid method (uncertainty in maximum built heights modeled using probability distributions; uncertainty in distribution parameters modeled using possibility distributions) could do better

  23. 4. Conclusion • Efficient testing approach for methods for decision under uncertainty • Uses real life data • Allows testing methods on many decisions and evaluate consequences for many outcomes of uncertain events • Abundant easily accessible databases from different fields • Can learn useful lessons from tests • Inflation of probability distributions counterproductive. Opposite effect on probabilistic and possibilistic optimum decision. • Identified weakness in possibilistic approach by accentuating difference between probability and possibility. Probability focuses on hazard easier to control; possibility does not. Cause: non additivity of possibility. • Small modeling errors can offset natural advantage of probability that it maximizes right objective function.

More Related