1 / 29

Statistical Comparison of Two or More Systems

Statistical Comparison of Two or More Systems. The most relevant of all the Basic Theory Lectures. No Holidays. THE MISSION. Your analysis task involves manipulating conditions of the system of interest from a prescribed set of options.

johnna
Download Presentation

Statistical Comparison of Two or More Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical Comparison of Two or More Systems The most relevant of all the Basic Theory Lectures. No Holidays.

  2. THE MISSION • Your analysis task involves manipulating conditions of the system of interest from a prescribed set of options. • Design of Experiments: Determine if the different options are really different. Is the best one really statistically better? • Ranking and Selection: What’s the probability that the best sample indicates the best system setting?

  3. VOCABULARY • Factor • An element of the system that will be manipulated • Setting or Level • A value that a Factor may assume

  4. EXAMPLE : Simulation model of Football (EA Sports) • Factors • Quarterback • Running Back • Strong Safety • Settings or Levels for Quarterback • Dante’ • Bret • Johnny U.

  5. TYPES OF DESIGNS • One Factor, Two Settings • Paired samples • Behrens-Fischer • Question: Which is Best? • More than one Factor • Factorial Designs • Partially Exhaustive Designs • Question: Are the settings significant difference-makers?

  6. PAIRED SAMPLES • Example: Quarterback Controversy! • Simulate St. Louis Rams vs. Tampa Bay Bucs, recording the Quarterback Rating • Level 1: Curt Warner • Level 2: Mark Bulger • Run the simulation 28 times for each player, resulting in data set • W1, W2, ..., W28 • B1, B2, ..., B28 • Is E[B] > E[W]?

  7. BRUTE FORCE • Confidence interval on the quantity E[W]-E[B] • If it doesn’t include 0.0, we have conclusive evidence that there is a difference • Equivalent to the Hypothesis Test • H0: E[B]=E[W]

  8. CALCULATIONS ON VARIANCES: SOME BASICS • Let X and Y be random variables

  9. CALCULATIONS ON VARIANCES: SOME BASICS • Let X and Y be random variables COV=0 if X and Y are independent.

  10. SAMPLE MEAN

  11. CONFIDENCE INTERVAL • a/2 probability of Type I error on each end of the confidence interval • basic interval for X-bar is

  12. BASIC CONFIDENCE INTERVAL

  13. SPREADSHEET HIGHLIGHTS 1 • (U-0.5)*SQRT(12) • zero mean • unit stddev • m + (U-0.5)*SQRT(12)*s • mean m • stddev s • uniform over an interval centered at m and s*SQRT(12)/2 wide

  14. COMMON RANDOM NUMBERS • Correlation is not always BAD! • Suppose we could INDUCE CORRELATION between the W’s and the B’s without adding any bias? • Reduces the theoretical variance of W-bar – B-bar • FREE POWER (the probability of correctly rejecting H0: equal means)

  15. STREAMING • Segregate the random number generation task into streams connected to phenomena Zi=aZi-1 mod m seed1 seed2 Inter-arrival times Service times 1. Change features of the service. 2. Use exact same arrival stream for comparing each service setting.

  16. SPREADSHEET HIGHLIGHTS 2 • Use same results of RAND() for building • Bulger samples • Warner samples • Note CI shrinkage • Try with identical sigma • Discuss “Estimation”

  17. Behrens-Fischer Problem • Comparison of Means • No pairs, equal sample sizes, or equal variances • Remember that we are after the variance of W-bar – B-bar • Common use: New samples vs. History

  18. SPREADSHEET HIGHLIGHTS

  19. MULTI-SETTING CASE • Can involve many Factors or just one • Treatment i has mean mi • Analysis of Variance (ANOVA) • Data from treatment 1, 2, ..., n • H0: m1 =...mn-1 =mn • Are the treatments distinguishable?

  20. DESIGN OF EXPERIMENT Determine Factors and Settings State Conclusion Design = Which Factors, Which Settings for each Treatment Collect Data According to Design Perform ANOVA

  21. FULL FACTORIAL • Build sample of All Combinations • Factors • Quarterback (2) • Running Back (3) • Strong Safety (3) • 2x3x3=18 Treatments

  22. HOW ANOVA WORKS • Xi,j is ith sample from jth treatment point • Assumed iid Normal (never!) • Decomposition of variability • Observation (Obs) • Treatment vs. Grand Mean (Tr) • Within Treatment (Res)

  23. HYPOTHESIS H0 • The treatment variability is random variability • The size of the treatment variability is in-scale with the residual variability • ANOVA uses sums of squares • g treatments • nt samples from treatment t

  24. ANOVA TABLE degrees freedom

  25. REMEMBER chi-SQUARED?From our Goodness-of-Fit Test • X~N(0,1) • for n independent X’s • sum of n X2 is chi-SQUARED with n degrees of freedom • if estimates (X-bar, sigma) were used to make X’s N(0,1), lose one d.f. per estimate

  26. F-distribution • X is chi-sq with n d.f. • Y is chi-sq with m d.f. • (X/n)/(Y/m) has F distribution

  27. ANOVA HYPOTHESIS TEST The normalizing s cancels!

  28. ANOVA HYPOTHESIS TEST • Compare the test statistic to a table • Reject if its big and conclude that ... • the Treatments are Different!

  29. SPREADSHEET HIGHLIGHTS

More Related