1 / 13

Module 3: Using Statistics for Benchmarking and Comparison

Module 3: Using Statistics for Benchmarking and Comparison. Common Measures Training Chelmsford, MA September 28, 2006. Comparison: Two Questions. Comparing two samples of one population, over time Are we confident performance changed? How much did performance change?

sauda
Download Presentation

Module 3: Using Statistics for Benchmarking and Comparison

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Module 3: Using Statistics for Benchmarking and Comparison Common Measures Training Chelmsford, MA September 28, 2006

  2. Comparison: Two Questions • Comparing two samples of one population, over time • Are we confident performance changed? • How much did performance change? • Comparing two samples from two different states • Are we confident performance is different? • How much difference is there?

  3. Key Concepts for Comparison • "Statistical Significance" / Significant Difference • Confidence Interval of a Difference • Minimum Detectable Difference

  4. “Statistical Significance” • Important ERP Question: Did facility performance change between the baseline and the post-certification inspections? • Population improvement? If we observed an improvement between the first sample and the second sample, are we confident there was an improvement in the overall population? • Remember, both the first and second samples’ point estimates have error associated with them

  5. “Statistical Significance” • “Statistical significance” means we are confident there was an improvement in the population • Other differences. Concept also applies when comparing results from two states • Are they significantly different?

  6. Understanding Differences • Difference = proportion or quantity. E.g., • Proportion: • 40% compliance in Round 1 • 60% compliance in Round 2 • Difference = 20% • Mean: • 300 pounds average waste generation in State A • 400 pounds average waste generation in State B • Difference = 100 pounds • Outcome measures, indicator scores and certification accuracy can have differences, too

  7. Confidence Intervals for Differences • Just like single sample observations, observed differences have error associated with them • Confidence interval reflects that error. E.g., • We are 95% confident that compliance increased 20% (+/-7%) • We are 95% confident that average waste generation is 75-125 pounds lower in State A than in State B

  8. Is the Difference Significant? Two ways to understand if the difference is significant • Formal hypothesis test • Less flexible • Test must be defined pre-sample • Or confidence interval of the difference…

  9. Confidence Interval of the Difference Confidence interval of the difference contains zero? • Difference is 10% +/- 7% (95% conf.) • 95% confidence that difference is between 3% and 17% (excludes zero) • Difference is statistically significant • Difference is 10% +/- 11% (95% conf.) • 95% confidence that difference is between -1% and 21% (could be zero) • Difference is not statistically significant

  10. Minimum Detectable Difference = How large a change in performance you need to see for it to be “statistically significant” • Important concept in sample planning • Depends on population size and sample size for each sample • Smaller detectable difference = larger sample size

  11. Minimum Detectable Difference • Think about what level of change would drive decision-making • Worthwhile to design samples to detect a 2% change? • Is it different for practical purposes? • Would you do anything differently than if you thought there was no change?

  12. Tour of the Spreadsheet Tools Two-sample sheets in Sample Planner and Results Analyzer.

  13. For more information… Contact Michael Crow • E-mail: mcrow@cadmusgroup.com • Phone: 703-247-6131

More Related