California Educational Research Association Anaheim, California December 1, 2011

1. California Educational Research Association Anaheim, California December 1, 2011 Effective Use of Benchmark Test and Item Statistics and Considerations When Setting Performance Levels

2. Objective Extend knowledge of assessment team to: Better understand test reliability and the influences of test composition and test length. Better understand item statistics and use them to identify items in need of revision Review of Benchmark Test and Item Statistics

3. Reliability is a measure of the consistency of the assessment Types of reliability coefficients (always range from 0 to 1) Test-retest Alternate forms Split-half Internal consistency (Cronbach’s Alpha/KR-20)

4. Reliability Influenced by Test Length • Spearman-Brown formula estimates reliabilities of shorter tests • Remember: The reliability of a score is an indication of how much an observed score can be expected to be the same if observed again. NOTE: See handout from STAR Technical Manual for exact cluster reliabilities.

5. Reliability Influenced by Test Length • Example: given a 75 item test with r=.95 • 40 item test has r=.91 • 35 item test has r=.90 • 30 item test has r=.88 • 25 item test has r=.86 • 20 item test has r=.84 • 10 item test has r=.72 • 5 item test has r=.56 NOTE: See handout from STAR Technical Manual for exact cluster reliabilities.

6. Reliability Statistics for CST’s(see handout) • Note that CST reliabilities range from .90 to .95 • Note that cluster reliabilities are consistent with those predicted by Spearman-Brown formula

7. Validity is the degree to which the test is measuring what was intended Types of test validity A. Predictive or Criterion (How does it correlate with other measures?) B. Content • How well does the test sample from the content domain? • How aligned are the items with regard to format and rigor

8. Validity Is Influenced by Reliability • Impact of Lower Reliability on Validity • Remember: Validity is the agreement between a test score and the quality it is believed to measure • Upper limit on validity coefficient is the square root of the reliability coefficient • 75 item test = square root of .95 = .97

9. Validity Is Influenced by Reliability • Upper limit on validity coefficient is the square root of the reliability coefficient • 75 item test =square root of .95=.97 • 30 item test= square root of .88=.94 • 20 item test= square root of .86=.93 • 10 item test = square root of .72=.85 • 5 item test = square root of .56=.75

10. Coefficient of Determination (R squared) • Square of validity coefficient gives “proportion of variance in the achievement construct accounted for by the test” • 75 item test =.97 squared=.94 • 30 item test=.94 squared=.88 • 20 item test=.93 squared=.86 • 10 item test=.85 squared=.72 • 5 item test=.75 squared=.56

11. Using Item Statistics (p-value & point-biserials) • Apply item analysis statistics from assessment reporting system(e.g. Datadirector, Edusoft, OARS, EADMS, etc.) • P-values (percent of group getting item correct • Most should be between 30 and 80 • Very high indicates it may be too easy; too low may indicate a problem item • Point-biserials (correlation of item with total score) • Most should be .30 or higher • Very low or negative generally indicates a problem with the item

12. Item statistics for CST’s(see handout) • Note that the range of P-values is consistent with most being between .30 and .80 • Note that median point-biserials are generally in the 40’s

14. Algebra 1

15. Algebra 1

16. Algebra 2

17. Geometry

18. Objective Extend knowledge of assessment team to: Better understand how performance level setting is key to predictive validity. Better understand how to create performance level bands based on equipercentile equating Maximizing Predictive Accuracy of District Benchmarks

19. Common Methods for Setting Cutoffs on District Benchmarks: Use default settings on assessment platform (e.g. 20%, 40%, 60%, 80%) Ask curriculum experts for their opinion of where cutoffs should be set Determine percent correct corresponding to performance levels on CSTs and apply to benchmarks Comparing District Benchmarks to CST Results

20. There is a better way! Comparing District Benchmarks to CST Results

21. “Two scores, one on form X and the other on form Y, may be considered equivalent if their corresponding percentile ranks in any given group are equal.” (Educational Measurement-Second Edition, p. 563) Comparing District Benchmarks to CST Results

22. Equipercentile Method of Equating at the Performance Level Cut-points Establishes cutoffs for benchmarks at equivalent local percentile ranks as cutoffs for CSTs By applying same local percentile cutoffs to each trimester benchmark, comparisons across trimesters within a grade level are more defensible Comparing District Benchmarks to CST Results

23. Equipercentile Equating MethodStep 1-Identify CST SS Cut-points

24. Equipercentile Equating Method Step 2 - Establish Local Percentiles at CST Performance Level Cutoffs (from scaled score frequency distribution)

25. Equipercentile Equating Method Step 3 – Locate Benchmark Raw Scores Corresponding to the CST Cutoff Percentiles (from benchmark raw score frequency distribution)

26. Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –Old Cutoffs

27. Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – Old Cutoffs

28. Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – Old Cutoffs

29. Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – New Cutoffs

30. Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –New Cutoffs

31. Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –New Cutoffs

32. Example: Classification AccuracyBiology

33. Example: Classification AccuracyBiology

34. Example: Classification AccuracyChemistry

35. Example: Classification AccuracyEarth Science

36. Example: Classification AccuracyPhysics

37. Things to Consider Prior to Establishing the Benchmark Cutoffs • Will there be changes to the benchmarks after CST percentile cutoffs are established? • If NO then raw score benchmark cutoffs can be established by linking CST to same year benchmark administration (i.e. spring 2011 CST matched to 2010-11 benchmark raw scores) • If YES then wait until new benchmark is administered and then establish raw score cutoffs on benchmark • How many cases are available for establishing the CST percentiles? (too few cases could lead to unstable percentile distributions)

38. Things to Consider Prior to Establishing the Benchmark Cutoffs (Continued) How many items comprise the benchmarks to be equated? (as test gets shorter it becomes more difficult to match the percentile cutpoints established on the CST’s)

39. SummaryEquipercentile Equating Method Method generally establishes a closer correspondence between the CST and Benchmarks When benchmarks are tightly aligned with CSTs, the approach may be less advantageous (i.e. elementary math) Comparisons between benchmark and CST performance can be made more confidently Comparisons between benchmarks within the school year can be made more confidently

40. Coming Soon from Illuminate Education, Inc.! Reports using the equipercentile methodology are being programmed to: (1) establish benchmark cutoffs for performance bands (2) create validation tables showing improved classification accuracy based on the method

41. Contact: Tom Barrett, Ph.D. President, Barrett Enterprises, LLC Director, Owl Corps, School Wise Press 2173 Hackamore Place Riverside, CA 92506 951-905-5367 (office) 951-237-9452 (cell)