1 / 13

ETM 607 – Output Analysis: Estimation of Relative Performance

ETM 607 – Output Analysis: Estimation of Relative Performance. Output comparison between two or more alternative systems Common Random Numbers (CRN) Comparison of several systems Optimization via simulation. ETM 607 - Comparing Alternatives.

jabari
Download Presentation

ETM 607 – Output Analysis: Estimation of Relative Performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ETM 607 – Output Analysis: Estimation of Relative Performance • Output comparison between two or more alternative systems • Common Random Numbers (CRN) • Comparison of several systems • Optimization via simulation

  2. ETM 607 - Comparing Alternatives • Usually, simulation is used for more than just a single model “configuration” • Often want to compare alternatives, select or search for the best (via some criterion) • Simple processing system: What would happen if the arrival rate were to double? • Cut interarrival times in half • Rerun the model for double-time arrivals • Make five replications

  3. ETM 607 – Simple Processing System Single Server Queue Arrival Rate ~Exp Service Rate ~ Norm • Performance Measures: • Average queue size • Max queue size • Average waiting time • Max waiting time • Average time in system • Max time in system • Server utilization • Number served

  4. ETM 607 - Results: Original vs. Double-Time Arrivals • Original – circles • Double-time – triangles • Replication 1 – filled in • Replications 2-5 – hollow • Note variability • Danger of making decisions based on one (first) replication • Hard to see if there are really differences • Need: Statistical analysis of simulation output data

  5. ETM 607 - Comparing Two Alternatives • General Idea: Approach I • Select some primary performance measure for comparison (e.g. production rate, average time in system, server utilization, etc…), denoted as Qi for alternative i. • Run alternative i for Rireplications to find the mean, standard deviation and confidence interval for performance parameter Qi. • If the CI’s of each alternative overlap, you cannot conclude there is statistical difference between the alternatives. • What if you increased the number of replications? • What if you changed the significance level (a) of the CI?

  6. ETM 607 - Comparing Two Alternatives • General Idea: Approach II • Select some primary performance measure for comparison (e.g. production rate, average time in system, server utilization, etc…), denoted as Qi for alternative i. • Run alternative i for Rireplications to obtain the performance parameter Qri, where r refers to the replication. • Find the difference between the performance measure for each replication (Qr1 – Qr2). • Compute the mean, standard deviation and CI for (Qr1 – Qr2). • If the CI of (Qr1 – Qr2) contains the value 0, then there is no statistical difference between the two alternatives.

  7. ETM 607 - Comparing Two Alternatives Perform In-Class Lab Exercise 11, Comparison of Two Alternatives

  8. ETM 607 - Comparing Two Alternatives: Common Random Number (CRN) • Step 4 of Lab 11 was possible because the same random number stream was used to compare each alternative. (The only change between both systems was the arrival rate). • This technique is referred to as Common Random Numbers (CRN). • Usually possible only when changing parameter values between alternatives. If logic changes, or the execution of random numbers, then this technique is not valid.

  9. ETM 607 - Comparing Two Alternatives: Common Random Number (CRN) cont. • For CRN case: where for differences and

  10. ETM 607 - Comparing Two Alternatives: Common Random Number (CRN) cont. • For more general case when CRN not valid: where And rounded to integer

  11. ETM 607 – Comparison of Several System Designs • How do you compare multiple system designs? • Depends on the goal: • Estimation of the parameter, Qifor alternative i. • Comparison of Qi to some control Q1which might be an existing system. • All pairwise comparisons Qr1 – Qr2. • Selection of the best Qi. • Goals 1,2, 3 all use CI’s. Goal 4 requires a type of optimization approach. • Performing analysis on multiple alternatives is somewhat laborious, but some simulation packages facilitate this process. See In-class Lab 11a.

  12. ETM 607 – Optimization via Simulation • Goal 4 was to find the best Qi. • What if there are many (thousands or millions) of alternatives to choose from? • One approach is to use meta-heuristics. • Register for ETM 645 to learn more about meta-heuristics. • Attend the last lesson to observe an application of optimization via simulation in the scheduling of Radar Warning Receivers. • Arena comes with a tool called “OptQuest”. Download via Google.

  13. ETM 607 – Optimization via Simulation Perform InClass Lab 11a – Comparison of multiple alternatives

More Related