1 / 13

Analysis of IAS Merit System Changes Post-2008

Detailed analysis of changes to the IAS Merit System post-2008, including statistical evidence, proposed changes, test variability, and implications for candidates. Regression analysis provided in Excel file.

miriamalfred
Download Presentation

Analysis of IAS Merit System Changes Post-2008

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ISM IAS Merit Calculation Observations June 24, 2010

  2. Summary of Observations • There is no Statistical Evidence that IAS Severity for RO 830-2 Post 2008 is Different from Pre 2008 • There is no Statistical Evidence that IAS Severity for RO 830-2 Post 2008 is Different from First 7 Tests • IAS Merits for 830-2 are Estimated to be Higher Post 2008 • Most Proposed Changes to the Merit System for IAS Turn the Cummins ISM into a Completely Different Test for 830-2 • Statistically Significant Increase in Test Variability • Statistically Significant Increase in IAS merits

  3. Summary of Observations • June 22, 2010 Proposed Changes Turn Candidates with 43<IAS<49 and with Previous Merits>1000 into Automatic Fails • June 22, 2010 Proposed Changes Turn Candidates with 31<IAS<43 and with Previous Merits=1000 into Fails • While it Appears that an Oversight May Have Been Made in the Derivation of IAS Min, Max and Anchor, Oversights and/or Mistakes Cannot be Retroactively Corrected Because Changing the Stake in the Ground Changes the Test • This is True for References and Candidates • It is Permissible to Correct References and Candidates Through the Use of Correction Factors and Severity Adjustments, but Not by Changing Established Targets

  4. Data Analysis • 40 Reference Test Results on RO 830-2 • Statistical Comparisons Must Take Lab into Account • 40 Reference Tests Divided into 3 Time Periods • Time Period 1: First 7 Tests (Pre-Severity Change) • Time Period 2: 18 Test Results Between 7/9/2005-9/6/2007 • Time Period 3: 15 Test Results Between 2/3/2008-5/10/2010 • 3 (5) Merit Proposals Evaluated for IAS • Merits1: Original: 16/27/49 • Merits2: 5/26/2010: 23/27/49 • Merits3: 6/22/2010: 22/31/43 • + 2 Bonus Merit System Evaluations

  5. Analysis Logic • IAS for Time Periods 1 and 2 are Always in Original Merits (Data as Presented in TMC Database) • IAS for Time Period 3 is Calculated Using All 3 Merit Proposals • Test the Null Hypothesis that IAS Merits are Equal for all 3 Time Periods • Merit Proposals in which the Null Hypothesis is Rejected are Indications of Changing the ISM Test for 830-2 • The Conclusion that Merits3 Changes the ISM Test for Candidates with IAS Above 31 is Just a Matter of Logic

  6. Simple Dotplots • Dotplots Do Not Take into Account Lab Effects and are Simply for Observational Purposes • Dotplots Show, Observationally, that the IAS Merits are Increasingly Mild when Moving from Merits1 to Merits2 to Merits3 • This is especially apparent when one considers the first test result, generated in Lab D and way out to the right (mild) in Time Period 1, to be an outlier • Merits4 is not as bad as Merits2 and Merits3 • Merits4: 20/27/49

  7. Statistical Models • IAS Merits = f (Lab, Time Period) • Test HO that All 3 Time Periods Have Equal IAS Severity • Only Time Period 3 is Adjusted by Merit Proposals • Statistical Evidence that Time Period 3 is Different is, Therefore, Not Good News for the Merit Proposal • Statistical Evidence that Time Period 3 has Higher Merits than Time Period 1 or 2 for Merits3 • Statistical Evidence that Time Period 3 has Higher Merits than the Aggregate of Time Periods 1 and 2 for Merits2 • Regression Analysis Attached in Excel File • LSM Results Graphed

  8. Graphs of Least Squares Means • The Least Squares Means for Each Time Period are Plotted for Each Merit System Proposal • Note that there is Only No Statistical Evidence of an Impact on ISM IAS Severity for Merits4 • Merits4 also Represents the Smallest Erosion in Test Precision

  9. What About Another Option, Merits5? • Based on Merits3 • Increase the Max from 43 Back to 49 • Merits5: 22/31/49 • Solves the Problem of Candidates Over 31, BUT • IAS Even More Mild! NOT GOOD

  10. Wait! I Want to Do Something • Sorry, Whatever you do to the Merit System Will Change the Test Severity and Precision from its Original Form

  11. Appendix • Data and Regression Analysis

More Related