Methods for Setting Cut Scores to Represent College Readiness

1. Methods for Setting Cut Scores to Represent College Readiness Presented To the Fall 2011 Meeting Of the Michigan Educational Research Association May 17, 2011

2. Resources • Contributors • MDE Measurement Research & Psychometrics Unit • ACT Research & Development Unit • National Center for Educational Achievement • Michigan Technical Advisory Committee

3. Defining College Readiness • ACT Methodology • Good basis for a starting point • What score on the ACT gives an 11th grader a 50/50 chance of obtaining a B or better in a… • first-semester of freshman year • credit-bearing (non-remedial) • …course in a related subject?

4. Defining College ReadinessACT Methodology • ACT Methodology for identifying College Ready Benchmarks • http://www.act.org/research/researchers/reports/pdf/ACT_RR2005-3.pdf • A logistic regression methodology identifying the point on the ACT subject score scale where a student has a 50% or better probability of achieving a B or better in the first-semester, credit-bearing, adequately related, freshman course • Start with data from Michigan Public Institutions of Higher Education • Identify appropriate credit-bearing freshman courses against which to analyze the relationship between MME scores and course grades • Analyze proportion of students receiving a B or better against MME scale scores

5. Defining College ReadinessACT Methodology Each dot represents a group of students with a specific test score, plotting the test score against the percent of students with that test score who earned a B or better

6. Defining College ReadinessACT Methodology The curved line is the logistic regression line of best fit through the cloud of points

7. Defining College ReadinessACT Methodology Horizontal line represents a 50% probability of earning a B or better

8. Defining College ReadinessACT Methodology Vertical line represents the test score that give a 50% probability of earning a B or better

9. Defining College ReadinessACT Methodology • Issues With the ACT Methodology for the Purpose of Setting Cut Scores on the MME • Non-symmetric, regression-based method • Focused on probability prediction • ACT’s purpose • Appropriate for ACT’s purpose • Identifying equivalent points on different scales requires the use of a symmetric procedure. • Using a non-symmetric procedure can result in a biased cut score because of regression to the mean when identifying a cut score not at the state average

10. Defining College ReadinessProposed Methodology • Proposed Methodology • A Method Based on Signal Detection Theory (SDT) • A symmetric method • Focused on maximizing classification consistency • Maximizes the percentage of students who are both • Classified as proficient on 11th grade MME • Achieved a B or better in their first-semester, freshman-year, credit-bearing college course • Classification consistency is the core of identifying both college readiness in high school and being on track in lower grades • Results in an unbiased cut score because regression to the mean is not an issue

11. Defining College ReadinessProposed Methodology • SDT in a nutshell, for our purposes • Focused on the ability to… • Make correct decisions when data contain both “signal” and “noise” • Identify thresholds of “signal” in the data that maximize the ability to make correct classification decisions

12. Defining College ReadinessProposed Methodology • Other uses of SDT-based methods germane to our purposes • Maximally accurate detection of… • Medical anomalies in reading radiological reports • Severe cases in Emergency Room triage • Medical impacts in clinical drug trials • Objects in naval sonar, military radar, and civilian air traffic control • Impact of interventions on memory and cognition • Our purpose • Maximally accurate detection of… • College readiness on MME • Being on-track to college readiness on MEAP

13. Defining College ReadinessProposed Methodology • Basic application of SDT-based methods for our purposes • Using the known outcomes of college course grades, set the MME cut score to… • Maximize the number of students who are consistently classified • Minimize the number of students who are inconsistently classified

14. Defining College ReadinessProposed Methodology • Start with data from Michigan Public Institutions of Higher Education • Identify appropriate first-semester, freshman, credit-bearing (non-remedial) courses against which to analyze the relationship between MME scores and course grades • Identify students who took both the MME as 11th graders and took those courses • Split those students into those who received a B or higher and those who received a B- or lower • Depicted graphically in the next slides

15. Each dot on the left side of the vertical line represents one student who achieved a B- or lower in a first-semester freshman year, credit-bearing course Each dot on the right side of the vertical line represents one student who achieved a B or higher in a first-semester freshman year, credit-bearing course

16. Each dot on the upper side of the horizontal line represents one student who achieved proficiency on the MME Each dot under the horizontal line represents one student who was not proficient on the MME

17. Inconsistent Proficient in 11th grade B- or lower in college Consistent Proficient in 11th grade B or higher in college Consistent Not proficient in 11th grade B- or lower in college

20. 75.00% Consistent

21. 79.69% Consistent

22. Defining College ReadinessProposed Methodology • Intend to create three cut scores based on this methodology • Produces four performance levels • All three cut scores can have externally validated meaning: • Example Cut Score Set 1: • A or better in college course as Advanced • B or better in college course as Proficient • C or better in college course as Partially Proficient • Example Cut Score Set 2: • B or better in selective enrollment 4-year Universities as Advanced • B or better in non-selective 4-year Universities as Proficient • B or better in community college as Partially Proficient • Graphical example shows one cut at a time being set • For convenience • Statistical methodology can do all three cuts simultaneously

23. Defining On-Track for the Highest Grade Level of Each MEAP Subject • From the new MME cut scores, identify new cut scores for MEAP as follows… • Mathematics – grade 8 • Reading – grade 8 • Science – grade 8 • Social Studies – grade 9 • Use test scores of students who took both the MME in Spring 2010 and the grade 9 MEAP in Fall of 2007 for Social Studies • Use test scores of students who took both the MME in Spring 2010 and the grade 8 MEAP in Fall of 2006 for Mathematics, Reading, and Science

24. Defining On-Track for the Highest Grade Level of Each MEAP Subject Each dot on the graph represents the grade 11 MME score and the score on the highest grade level of MEAP for an individual student

25. Defining On-Track for the Highest Grade Level of Each MEAP Subject MME cut scores previously identified using SDT-based methods

26. Defining On-Track for the Highest Grade Level of Each MEAP Subject Partially Proficient on MME Not Proficient on MME Proficient on MME Advanced on MME

27. Defining On-Track for the Highest Grade Level of Each MEAP Subject MEAP grade 8 cut scores to be identified

28. Defining On-Track for the Highest Grade Level of Each MEAP Subject

29. Defining On-Track for the Highest Grade Level of Each MEAP Subject Adjust the Grade 8 MEAP Cut Scores Up or Down to Obtain Maximum Classification Consistency

30. Defining On-Track for the Remaining Grade Levels of Each MEAP Subject • Use test scores of students who took MEAP in both Fall 2009 and Fall 2010 for Reading and Math • Use test scores of students who took MEAP in both Fall 2007 and Fall 2010 for Science and Social Studies • Why not use the same cohort all the way back to the lowest MEAP grade? • Data are not available • Need to use the most recent data for identifying each grade level’s cut score • Need to reflect the current state of Michigan education

31. Defining On-Track for the Remaining Grade Levels of Each MEAP Subject • Use same SDT-based method to go systematically down the line one grade at a time • Mathematics and Reading • Connect grade 8 to grade 7 • Connect grade 7 to grade 6 • Connect grade 6 to grade 5 • Connect grade 5 to grade 4 • Connect grade 4 to grade 3 • Science • Connect grade 8 to grade 5 • Social Studies • Connect grade 9 to grade 6

32. Next Steps • Obtain data from Institutions of Higher Education (currently being done, from CEPI) • Determine the appropriate credit-bearing freshman courses against which to analyze MME scores (currently being done, with ACT and CEPI) • Identify college-ready cut scores for all subjects on the MME • Identify on-track cut scores for all subjects on the MEAP • Recommend to the State Board of Education the final cut scores for Fall 2011 MEAP and Spring 2012 MME

33. References Boutis, K. , Pecaric, M. , Seeto, B. and Pusic, M. (2010). Using signal detection theory to model changes in serial learning of radiological image interpretation. Advances in health sciences education: theory and practice, 15(5), 647-58. Despins, L., Scott-Cawiezell, J., and Rouder, J. (2010). Detection of patient risk by nurses: a theoretical framework. Journal of advanced nursing, 66(2), 465-74. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley. Klein, S. and Levi, D. (2009) Stochastic model for detection of signals in noise. Journal of the Optical Society of America: Optics, image science, and vision, 26(11),110-26. Merlo-Pich, E. and Gomeni, R. (2008). Model-based approach and signal detection theory to evaluate the performance of recruitment centers in clinical trials with antidepressant drugs. Clinical pharmacology and therapeutics, 84(3), 378-84. Neal, A. and Kwantes, P. (2009) An evidence accumulation model for conflict detection performance in a simulated air traffic control task. Human factors, 51(2), 164-80. Pigeau, R., Angus, R., O’Neill, P. and Mack, I. (1995). Vigilance latencies to aircraft detection among NORAD surveillance operators. Human Factors, 37(3), 622-634.

34. Contact Information • Joseph A. Martineau, Ph.D. • Executive Director • Bureau of Assessment & Accountability • Michigan Department of Education • martineauj@michigan.gov