Teaching Simulation

1. Teaching Simulation Roger Grinde, roger.grinde@unh.edu University of New Hampshire Files: http://pubpages.unh.edu/~rbg/TMS/TMS_Support_Files.html

2. Teaching Simulation • Do you teach simulation? • In which courses? • With spreadsheets? Add-Ins? • Monte Carlo? Discrete Event? • Do you use simulation to help teach other topics? • Do other courses at your school use simulation?

3. Session Overview • Common Student Misunderstandings • Simulation-Related Learning Goals • Motivations • Building on Other Methodologies • Effects of Correlation • Interpreting Results • Software Issues • Considerations, Recommendations

4. Student Misunderstandings • What are some misunderstandings students have about decision-making in the face of uncertainty? • What are some common errors students make in simulation?

5. Some Considerations • Decide which learning goals are most important, and structure coverage so those goals are attained. • Student backgrounds • Time constraints • Overall course objectives • Inter-course relationships, role of course in curriculum • Monte-Carlo and/or Discrete-Event? Related software selection question. • Teaching environment, class size, TA support, etc.

6. Learning Goals • What are your learning goals when teaching simulation? • Fundamental Concepts • Methodology of Simulation • Applications of Simulation • Modeling Knowledge & Skills • Critical & Analytical Thinking

7. Mapping: Learning Goals to Examples

8. Mapping: Goals to Examples

9. Motivations (Why is simulation useful?) • Two investment alternatives • A: Invest $10,000. • Probability of a $100,000 gain is 0.10 • Probability of a $10,000 loss is 0.90 • B: Invest $10,000 • Probability of a $500 gain is 1.0 • Which would you choose? • Why?

10. Risk-Informed Decision Making • Appropriate and inappropriate uses of averages. • Managers manage risk. • Simulation gives us a tool to help us evaluate risk. • Risk: The uncertainty associated with an undesirable outcome. • Risk is not the same as just being uncertain about something, and is not just the possibility of a bad outcome. • Risk considers the likelihood of an undesirable outcome (e.g., the probability) as well as the magnitude of that outcome.

11. “Flaw of Averages” (Sam Savage) • Article by Sam Savage (http://www.stanford.edu/~savage/faculty/savage/) • Annuity Illustration (historical simulation)

12. Fixed (Known) Inputs Outputs & Performance Measures Random (Uncertain) Inputs Simulation Model Decision Variables Simulation Model Schematic • Concept of an output “distribution.”

13. Foundations of Simulation • Randomness, Uncertainty • Probability Distributions • Tools • Dice Roller (John Walkenbach: http://www.j-walk.com/ss) • Die Roller (modified) • Interactive Simulation Tool

14. Extending Other Methodologies • Spreadsheet Engineering • Base Case Analysis • What-If Analysis, Scenario Analysis • Critical Value Analysis • Sensitivity Analysis • Simulation

15. Extending Other Methodologies • Familiar Example/Case; Students have already developed model and done some deterministic analysis. • Students provided with some probability distribution information • Develop comfort with mechanics of simulation • See the “value added” of simulation • Provides entry point for discussion of important questions

16. Example: Watson Truck • Adapted from Lawrence & Weatherford (2001) • Students have previously built base-case model, done “critical value” analysis (using Goal Seek), and have done sensitivity analysis (data tables, tornado charts) • Link to files: PDF, Sensitivity, Simulation

17. Watson Truck: Inputs

18. Watson Truck: Base Case Model

19. Watson Truck: Sensitivity Analysis

20. Watson: Simulation

21. Learning Goals Addressed (at least partially) • Linkage with other course/functional area • What inputs should we simulate? • Useful probability distributions. Choice of parameters. Subjective versus objective estimates. • Concept of an output distribution • What results are important? • Sources of error in simulation • Simulation mechanics • Simulation in context with other tools

22. Example: Single-Period Portfolio • Simple example, but helps address a number of learning goals • Do we need to simulate? • Effect of correlation among input quantities • Confidence vs. Prediction (certainty) intervals • Quantification of risk, multiple decision criteria • Optimization concepts within simulation context • Precision of estimates from simulation • Link to file

23. Spreadsheet

24. Do we need simulation? • Assuming we know the distributions for the returns, do we need simulation to compute the • expected return of the portfolio? • variance of the portfolio? • tail probabilities?

25. What if the asset returns are correlated? • What is the effect of correlation on the distribution of portfolio returns?

26. Results (n=1000) • No Correlation • Mean = $6842 • Standard Deviation = $5449 • 5% VaR = ($2165) • Positive Correlation • Mean = $6409 • Standard Deviation = $7386 • 5% VaR = ($5655)

27. Decision Criteria, Risk Measures • What criteria are important for making decision as to where to invest? Average? Standard Deviation? Minimum? Maximum? Quartiles? VaR? Probability of Loss? • Measures of risk. • Simulation gives us the entire output distribution. • Entry point for optimization within simulation context • Alternate scenarios, efficient frontier, OptQuest, RiskOptimizer, etc.

28. Confidence Intervals • Students can (usually) calculate a confidence interval for the mean. • Do they know what it means? • Reconciling confidence and prediction intervals.

29. Sample Results (Portfolio Problem) • 90% CI on Mean Dollar Return: ($6025, $6794) • What does that confidence interval mean? • Common (student) error • What does the CI about an individual outcome? For example, from this year’s return?

30. Sample Results (cont) • Cumulative Percentiles of the Portfolio Return Distribution • What do these results mean? • What is the 90% “prediction” (or “certainty”) interval (centered around the median)?

31. Putting Them Together • 90% Confidence Interval for the Mean • ($6025, $6794) • 90% Prediction Interval (centered around median) • (-$5655, $18,659) • Note: Crystal Ball uses the term “certainty”) • Students: • Understand the difference? • Understand when one is more appropriate than the other?

32. Precision of Simulation Results • Since we know the true value of the mean (for the portfolio problem), this can be a good example to look at precision and sample size issues. • Confidence interval for proportion or for a given percentile sometimes makes more sense.

33. Crystal Ball: Precision Control • Nice way to illustrate effect of sample size. • Precision Control stops simulation based on user-specified precision on the mean, standard deviation, and/or a percentile. • Actually, CB stops whenever the first of a number of conditions occurs (e.g., maximum number of trials, precision specifications). • Example (Portfolio Allocation) • Example (Option Pricing)

34. Precision: Portfolio Example

35. Precision: Option Pricing Example

36. Crystal Ball Functions and Simple VBA Control • Crystal Ball provides built-in functions • Distribution Functions (e.g., CB.Normal) • Functions for Accessing Simulation Results (e.g., CB.GetForeStatFN) • Control through VBA • For some students, can be a hook into greater interest in simulation and/or VBA/DSS. • Allows one to prepare a simulation-based model for someone who doesn’t know Crystal Ball. • Example

37. VBA-Enabled Example

38. CB. Functions and VBA • CB. Distribution Functions • e.g., CB.Normal, CB.Uniform, CB.Triangular) • CB. Functions for reporting results • CB.GetForeStatFN, CB.GetCertaintyFN, CB.GetForePercentFN • VBA: simple to automate specific processes Sub RunSimulation() CB.ResetND CB.Simulation Range("n_trials").Value End Sub Sub CreateReport() CB.CreateRpt ' CB.CreateRptND cbrptOK End Sub

39. Learning Goals Revisited • Decide which learning goals are the most important, and structure coverage so those goals are attained. • Student backgrounds • Time constraints • Overall course objectives • Mapping of learning goals to examples, cases, and projects that you will use.

40. Mapping: Learning Goals to Examples

41. Mapping: Possible Learning Goals to Examples

42. Common Student Errors • Thinking of simulation as the method of first choice. • Simulating too many quantities. • Too much focus on distribution/parameter selection or on the numerical results, not enough on insights/decision. • Misinterpretation of results, especially confidence intervals • Modeling: Using same return, lead time, etc. for every time period/order, etc. (difference between deterministic and simulation models) • Choosing the assumptions, distributions, parameters, etc. that give the “best” numerical results.

43. Software Issues: Monte-Carlo • Alternatives • “Full-Service” Add-In? (e.g., @Risk, Crystal Ball, XLSim by Sam Savage, RiskSim) • “Helper” Workbook? (e.g., Interactive Simulation Tool with Random Number Function support) • “Native” Excel? • All have advantages, disadvantages • Back to learning objectives, role of course, student audience, etc.

44. Software Issues: Discrete-Event • Alternatives • Stand-alone package (e.g., Arena, Process Model, Extend) • Excel Add-In (e.g., SimQuick by David Hartvigsen) • Native Excel modeling augmented by Monte Carlo tool (e.g., QueueSimon by Armann Ingolfsson) • DE Simulation can be a great way to help teach concepts in other areas (e.g., queuing, inventory) • Don’t necessarily need to teach DE Simulation to be able to use it to teach other things.

45. Other Considerations • Program-level, inter-course objectives • Role of course in curriculum • Level/background of students • Monte-Carlo and/or Discrete-Event? Related software selection question. • Teaching environment, class size, TA support, etc. • How much of course can/should be devoted to simulation?

46. Recommendations • Learning Goals: Figure out what you really want students to learn and be able to do, after your class is over; in other classes, internships, future jobs? How can simulation coverage help accomplish these goals? • Cases: Engage students in the business problem, let them discover relevance of simulation. • Student-Developed Projects: Students gain better awareness of all the “little” decisions involved in modeling and simulation.

47. Additional Slides

48. Concept Coverage Through Examples • Philosophy: Expose students to a number of application areas, but at the same time covering fundamental decision-making, modeling, and analysis concepts and methodologies. • Counter to the way many of us were taught. • Key: We need to clearly understand which concepts we’re trying to convey with each example.

49. Examples that Work Well • Fundamentals: Dice Roller, Interactive Simulation Tool • Personal Decisions:Car Repair/Purchase Decision, Portfolio (single period, based on CB Model), College Funding (based on Winston & Albright) • Capital Project Evaluation: Truck Rental Company (based on Lawrence & Weatherford), Project Selection/Diversification (CB Model), Product Development & Launch (CB Model) • Finance:Stock Price Models, Option Pricing, Random Walks, Mean Reverting Processes

50. Examples (continued) • Inventory:DG Winter Coats (NewsVendor), Antarctica (multi-period, based on Lapin & Whisler) • Queuing:QueueSimon (Armonn Ingolfsson) • Games/Tournaments, Sports:NCAA Tourney (based on Winston & Albright), Home Run Derby Baseball Simulation (VBA-enabled), Baseball Inning Simulation • Simulation in Teaching Other Topics:Revenue Management Illustration, QueueSimon (Armonn Ingolfsson) • Crystal Ball Features:CB Macros, CB Functions