1 / 16

Pitfalls

Pitfalls. No problem definition. "We have many problems in our line of business. Just start building a simulation model of our business while we figure out which questions we want to be answered.". When the questions arrive (if at all answerable by simulation), the model cannot be used.

dutch
Download Presentation

Pitfalls

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pitfalls

  2. No problem definition "We have many problems in our line of business. Just start building a simulation model of our business while we figure out which questions we want to be answered." When the questions arrive (if at all answerable by simulation), the model cannot be used. Too much details: complex simulation model many parameters to assess many errors to correct lengthy simulation runs Too few details: not enough information at best only partial answers

  3. One-sided problem definition "What investments do I have to make to keep my customers satisfied?" Clarify: possible investments, quality of service, assumptions about demand. Both client and consultant should understand and agree. understand make formal and explain! Problem definition = simulation contract.

  4. Hidden assumptions Many assumptions are made for a simulation study. Keep track of assumptions made. "In DCT case, total crane service time of an X-type truck succeeding an Y-type truck is normally distributed with average a(X,Y) and standard deviation s(X,Y)." If possible, validate (compare with recorded trace). In all cases: mention in report. Hidden assumptions: used in model but not reported. Hidden assumptions may lead to costly mistakes!

  5. Force fit the model Tandem queue arrival: expo(1), p1, p2: triangle(0.5,0.9,1.3) 10 subruns of 1000; initial run of 10 measured queue sizes p1: 1.71, p2: 0.42 First model: correct average but variance too small arrival: expo(1), p1, p2: triangle(0.7,0.8,0.9) measured queue sizes p1: 1.56, p2: 0.11 Fiddle with average to make model fit p1: triangle(0.71,0.81,0.91), p2: triangle(0.735,0.835,0.935) measured queue sizes p1: 1.78, p2: 0.42 Analyze 11% increased demand. 3.5/0.85 versus 4.6/3.9!

  6. Improper use of statistics “Giving Statistical Packages to Engineers is like Giving Guns to Children,” Dick Mensing "Statistics don't lie, but liars use statistics" Darell Huff

  7. Methodology pitfall Simulation = Experimentation. Design of experiments: established rules. Hypothesis before experiment. New hypothesis = new experiment. Hypothetical case: beer appreciation. Brewers have designed "X", a new low-cost beer. Hypothesis: appreciation of "X" not lower than other beers. Consumer appreciation is tested in various Dutch cities, relatively to "Amstel" and "Grolsch".

  8. Beer appreciation Table of average appreciations per city. city X Ams Grol A'dam 6.5 7.2 6.9 R'dam 7.1 6.8 7.3 den Haag 6.3 6.5 7.1 Utrecht 5.7 6.1 6.2 E'hoven 7.1 6.9 7.0 Enschede 5.9 6.6 7.5 Gr'ingen 6.2 6.3 6.5 Tilburg 6.9 7.0 6.7 Haarlem 6.8 7.2 6.8 Breda 7.2 6.9 7.2 Nijmegen 7.0 7.1 7.3 Statistical analysis: with 98% confidence, people from N-Brabant prefer (on average) "X" above the other two. Hypothesis rejected.

  9. Beer marketing strategy Market "X" as a local beer in N-Brabant. Call it "Brabobier". Make a lot of money. What is wrong? With 98% confidence, people from N-Brabant prefer "X" above the other two. New hypothesis based on sample observation. Confidence cannot be assessed. New, independent experiment is needed.

  10. Observe random sample Dice throwing experiment (Arena): 25 throws; sequence 2321116466156134222666665 five equals in a row! x(19)=...=x(24)=6: probability 0.00013 twenty possibilities for first: 0.0026 five equals (instead of sixes): 0.0156 Hypothesis: by throwing a dice 25 times, five equals in row will occur. Confirmed with 98.5% confidence (on given sample). Any recorded sample contains remarkable data. You need a second sample to confirm a hypothesis based on observing the first sample.

  11. Simulation analogon Problem in business process (e.g. DCT). Assess tentative solution through simulation. Hypothesis: performance indicator x after implementing the solution will be lower on average than in current situation. Boss needs 90% confidence in order to start implementing it. Indicator Average Half Width x 4.18 0.79 30 subruns of 1000 hrs current tentsol x 3.89 0.85 Test hypothesis: evaluate C result: 0.73 T

  12. Longer simulation Indicator Average Half Width x 4.27 0.41 30 subruns of 4000 hrs current tentsol x 3.92 0.47 hypothesis confidence: 0.87 Indicator Average Half Width x 4.32 0.38 current 30 subruns of 6000 hrs tentsol x 3.91 0.41 outcome: 0.93 ? conclusion: with 93% reliability solution gives lower average x

  13. Indicator Average Half Width x 4.32 0.38 current tentsol x 3.91 0.41 Persistent experimentation Repeat an experiment until it satisifies hypothesis. Statistically analyze each experiment individually. Hypothesis corroboration with incorrect confidence! After 3 experiments goal of 90% confidence attained. A new, independent simulation run is needed to confirm! Choose different initial value for random generator and repeat the last experiment.

  14. Cheating recipes Confirm hypothesis by simulation with 95% reliability. Do many independent simultation experiments. Only retain favorable ones. Claim you did not execute the unfavorable ones. Speeding up the process: Perform e.g. 50 iterations. Look for a most favorable sequence of 30, e.g. 7-36. Define iterations 0-6 as initial run. Perform 30 iterations. Round off favorably (model parameters and simulation results)

  15. Abuse of statistics Recent uncovered evidence shows that nurse Lucia de B has been convicted of murder on the basis of statistical arguments only and by abusing just about every rule in the book.  * Whether or not an incident was classified as suspect depended on whether or not Lucia was on duty (simply a question of checking which nurse is on duty and then asking enough doctors till you get a "suspicious" verdict. * Data was collected in this way till there was enough to condemn her. * A professor of statistics in law, and trained mathematician, does not know the meaning of one of the most basic statistical concepts - the p-value. (Statistician xxx multiplied three independent p-values in order to obtain a combined p-value).It somehow reminds me of the old method to see if someone is a witch - if they drown they were innocent, if they are guilty you can burn them.

  16. Quote "The only way to have real success in science...is to describe the evidence very carefully without regard to the way you feel it should be." Richard Feynman

More Related