Maximizing Predictive Accuracy of District Benchmarks

Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012 Maximizing Predictive Accuracy of District Benchmarks

Objective Better understand how performance level setting is key to predictive validity. Better understand how to create performance level bands based on equipercentile equating Maximizing Predictive Accuracy of District Benchmarks

Common Methods for Setting Cutoffs on District Benchmarks: Use default settings on assessment platform (e.g. 20%, 40%, 60%, 80%) Ask curriculum experts for their opinion of where cutoffs should be set Determine percent correct corresponding to performance levels on CSTs and apply to benchmarks Comparing District Benchmarks to CST Results

There is a better way! Comparing District Benchmarks to CST Results

“Two scores, one on form X and the other on form Y, may be considered equivalent if their corresponding percentile ranks in any given group are equal.” (Educational Measurement-Second Edition, p. 563) Comparing District Benchmarks to CST Results

Equipercentile Method of Equating at the Performance Level Cut-points Establishes cutoffs for benchmarks at equivalent local percentile ranks as cutoffs for CSTs By applying same local percentile cutoffs to each trimester benchmark, comparisons across trimesters within a grade level are more defensible Comparing District Benchmarks to CST Results

Equipercentile Equating MethodStep 1-Identify CST SS Cut-points

Equipercentile Equating Method Step 2 - Establish Local Percentiles at CST Performance Level Cutoffs (from scaled score frequency distribution)

Equipercentile Equating Method Step 3 – Locate Benchmark Raw Scores Corresponding to the CST Cutoff Percentiles (from benchmark raw score frequency distribution)

Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –Old Cutoffs

Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – Old Cutoffs

Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – New Cutoffs

Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –New Cutoffs

Example: Classification AccuracyBiology

Example: Classification AccuracyChemistry

Example: Classification AccuracyEarth Science

Example: Classification AccuracyPhysics

Things to Consider Prior to Establishing the Benchmark Cutoffs • Will there be changes to the benchmarks after CST percentile cutoffs are established? • If NO then raw score benchmark cutoffs can be established by linking CST to same year benchmark administration (i.e. spring 2011 CST matched to 2010-11 benchmark raw scores) • If YES then wait until new benchmark is administered and then establish raw score cutoffs on benchmark • How many cases are available for establishing the CST percentiles? (too few cases could lead to unstable percentile distributions)

Things to Consider Prior to Establishing the Benchmark Cutoffs (Continued) How many items comprise the benchmarks to be equated? (as test gets shorter it becomes more difficult to match the percentile cutpoints established on the CST’s)

SummaryEquipercentile Equating Method Method generally establishes a closer correspondence between the CST and Benchmarks Comparisons between benchmark and CST performance can be made more confidently Comparisons between benchmarks within the school year can be made more confidently

Coming Soon from Illuminate Education, Inc.! Reports using the equipercentile methodology are being programmed to: (1) establish benchmark cutoffs for performance bands (2) create validation tables showing improved classification accuracy based on the method

Contact: Tom Barrett, Ph.D. President, Barrett Enterprises, LLC 951-905-5367 (office) 951-237-9452 (cell) DrTBarrett@gmail.com

Maximizing Predictive Accuracy of District Benchmarks