1 / 33

Simulation Modeling and Analysis

Simulation Modeling and Analysis. Session 12 Comparing Alternative System Designs. Outline. Comparing Two Designs Comparing Several Designs Statistical Models Metamodeling. Comparing two designs. Let the average measures of performance for designs 1 and 2 be  1 and  2 .

karleigh-carpenter
Download Presentation

Simulation Modeling and Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Simulation Modeling and Analysis Session 12 Comparing Alternative System Designs

  2. Outline • Comparing Two Designs • Comparing Several Designs • Statistical Models • Metamodeling

  3. Comparing two designs • Let the average measures of performance for designs 1 and 2 be 1 and 2. • Goal of the comparison: Find point and interval estimates for 1 - 2

  4. Example • Auto inspection system design • Arrivals: E(6.316) min • Service: • Brake check N(6.5,0.5) min • Headlight check N(6,0.5) min • Steering check N(5.5,0.5) min • Two alternatives: • Same service person does all checks • A service person is devoted to each check

  5. Comparing Two Designs -contd • Run length (ith design ) = Tei • Number of replications (ith design ) = Ri • Average response time for replication r (ith design = Yri • Averages and standard deviations over all replications, Y1* = S Yri / Ri and Y2* , are unbiased estimators of 1 and 2.

  6. Possible outcomes • Confidence interval for 1 - 2 well to the left of zero. I.e. most likely 1 < 2. • Confidence interval for 1 - 2 well to the right of zero. I.e. most likely 1 > 2. • Confidence interval for 1 - 2 contains zero. I.e. most likely 1 ~ 2. • Confidence interval (Y1* - Y2*) ± t /2, s.e.(Y1* - Y2*)

  7. Independent Sampling with Equal Variances • Different and independent random number streams are used to simulate the two designs. Var(Yi*) = var(Yri)/Ri = i2/Ri Var(Y1* - Y2*) = var(Y1*) + var(Y2*) = 12/R1 + 22/R2= VIND • Assume the run lengths can be adjusted to produce 12 ~22

  8. Independent Sampling with Equal Variances -contd • Then Y1* - Y2* is a point estimate of 1 - 2 Si2 =  (Yri - Yi*)2/(Ri - 1) Sp2 = [(R1-1) S12 + (R2-1) S22]/(R1+R2-2) s.e.(Y1*-Y2*) = Sp (1/R1 + 1/R2)1/2  = R1 + R2 -2

  9. Independent Sampling with Unequal Variances s.e.(Y1*-Y2*) = (S12/R1 + S22/R2)1/2  = (S12/R1 + S22/R2)2/M where M = (S12/R1)2/(R1-1) + (S22/R2)2/(R2-1) • Here R1 and R2 must be > 6

  10. Correlated Sampling • Correlated sampling induces positive correlation between Yr1 and Yr2 and reduces the variance in the point estimator of Y1*-Y2* • Same random number streams used for both systems for each replication r (R1 = R2 = R) • Estimates Yr1 and Yr2 are correlated but Yr1 and Ys2 (r n.e. s) are mutually independent.

  11. Recall: Covariance var(Y1* - Y2*) = var(Y1* ) + var(Y2* ) - 2 cov(Y1* , Y2* ) = = 12/R + 22/R - 2 121 2/R = VCORR = VIND - 2 121 2/R Recall: definition of covariance cov(X1,X2) = E(X1 X2) - m1 m2 = = corr(X1 X2) s1 s2 = = r s1 s2

  12. Correlated Sampling -contd • Let Dr =Yr1 - Yr2 D* = (1/R)  Dr = Y1* - Y2* SD2 = (1/(R - 1))  (Dr - D*)2 • Standard error for the 100(1- )% confidence interval s.e.(D*) = s.e.(Y1* - Y2* ) = SD/ R (Y1* - Y2*) ± t /2, SD/ R

  13. Correlated Sampling -contd • Random Number Synchronization Guides • Dedicate a r.n. stream for a specific purpose and use as many streams as needed. Assign independent seeds to each stream at the beginning of each run. • For cyclic task subsystems assign a r.n. stream. • If synchronization is not possible for a subsystem use an independent stream.

  14. Example: Auto inspection An = interarrival time for vehicles n,n+1 Sn(1) = brake inspection time for vehicle n in model 1 Sn(2) = headlight inspection time for vehicle n in model 1 Sn(3) = steering inspection time for vehicle n in model 1 • Select R = 10, Total_time = 16 hrs

  15. Example: Auto inspection • Independent runs -18.1 < 1-2 < 7.3 • Correlated runs -12.3 < 1-2 < 8.5 • Synchronized runs -0.5 < 1-2 < 1.3

  16. Confidence Intervals with Specified Precision • Here the problem is to determine the number of replications R required to achieve a desired level of precision e in the confidence interval, based on results obtained using Ro replications R = (t /2,Ro-1 SD/e)2

  17. Comparing Several System Designs • Consider K alternative designs • Performance measure i • Procedures • Fixed sample size • Sequential sampling (multistage)

  18. Comparing Several System Designs -contd • Possible Goals • Estimation of each i • Comparing i to a control 1 • All possible comparisons • Selection of the best i

  19. Bonferroni Method for Multiple Comparisons • Consider C confidence intervals 1-i • Overall error probability E = j • Probability all statements are true (the parameter is contained inside all C.I.’s) P  1 - E • Probability one or more statements are false P E

  20. Example: Auto inspection (contd) • Alternative designs for addition of one holding space • Parallel stations • No space between stations in series • One space between brake and headlight inspection • One space between headlight and steering inspection

  21. Bonferroni Method for Selecting the Best • System with maximum expected performance is to be selected. • System with maximum performance and maximum distance to the second best is to be selected. i - max j ij 

  22. Bonferroni Method for Selecting the Best -contd 1.- Specify  ,  and R0 2.- Make R0 replications for each of the K systems 3.- For each system i calculate Yi* 4.- For each pair of systems i and j calculate Sij2 and select the largest Smax2 5.-Calculate R = max{R0, t2 Smax2 / 2} 6.- Make R-R0 additional replications for each of the K systems 7.- Calculate overall means Yi** = (1/R)  Yri 8.-Select system with largest Yi** as the best

  23. Statistical Models to Estimate the Effect of Design Alternatives • Statistical Design of Experiments • Set of principles to evaluate and maximize the information gained from an experiment. • Factors (Qualitative and Quantitative), Levels and Treatments • Decision or Policy Variables.

  24. Single Factor, Randomized Designs • Single Factor Experiment • Single decision factor D ( k levels) • Response variable Y • Effect of level j of factor D, j • Completely Randomized Design • Different r.n. streams used for each replication at any level and for all levels.

  25. Single Factor, Randomized Designs -contd • Statistical model Yrj =  + j + rj where Yrj = observation r for level j  = mean overall effect j = effect due to level j rj = random variation in observation r at level j Rj= number of observations for level j

  26. Single Factor, Randomized Designs -contd • Fixed effects model • levels of factors fixed by analyst • rj normally distributed • Null hypothesis H0: j = 0 for all j=1,2,..,k • Statistical test: ANOVA (F-statistic) • Random effects model • levels chosen at random • j normally distributed

  27. ANOVA Test • Levels-replications matrix • Compute level means (over replications) Y.i* and grand mean Y..* • Variation of the response w.r.t. Y..* Yrj - Y..* = (Y.j* - Y..*) + (Yrj - Y.j*) • Squaring and summing over all r and j SSTOT = SSTREAT + SSE

  28. ANOVA Test -contd • Mean square MSE = SSE/(R-k) is unbiased estimator of var(Y). I.e. E(MSE) = 2 • Mean square MSTREAT = SSTREAT/(k-1) is also unbiased estimator of var(Y). • Test statistic F = MSTREAT / MSE • If H0 is true F has an F distribution with k-1 and R-k d.o.f. • Find critical value of the statistic F1- • Reject H0 if F > F1-

  29. Metamodeling • Independent (design) variables xi, i=1,2,..,k • Output response (random) variable Y • Metamodel • A simplified approximation to the actual relationship between the xi and Y • Regression analysis (least squares) • Normal equations

  30. Linear Regression • One independent variable x and one dependent variable Y • For a linear relationship E(Y:x) = 0 + 1 x • Simple Linear Regression Model Y = 0 + 1 x + 

  31. Linear Regression -contd • Observations (data points) (xi,Yi) i=1,2,..,n • Sum of squares of the deviations i2 L =i2 =  [ Yi - 0’ - 1(xi - x*)]2 • Minimizing w.r.t 0’ and 1 find 0’* =  Yi /n 1*=  Yi (xi - x*)/  (xi - x*)2 0* = 0’* - 1*x*

  32. Significance Testing • Null Hypothesis H0: 1 = 0 • Statistic (n-2 d.o.f) t0 = 1*/(MSE/Sxx) where MSE = (Yi - Ypi)/(n-2) Sxx = xi2 - ( xi )2/n • H0 is rejected if |t0| > t/2,n-2

  33. Multiple Regression • Models Y = 0 + 1 x1 + 2 x2 + ... + m xm +  Y = 0 + 1 x+ 2 x2 +  Y = 0 + 1 x1 + 2 x2 + 3 x1 x2 + 

More Related