290 likes | 356 Views
Presentation to the Michigan State Board of Education September 13, 2011. Career- and College-Ready Cut Scores For MEAP and MME. ACT Research & Development Unit National Center for Educational Achievement Michigan Technical Advisory Committee MDE/BAA Measurement Research & Psychometrics Unit.
E N D
Presentation to the Michigan State Board of Education September 13, 2011 Career- and College-Ready Cut Scores For MEAP and MME
ACT Research & Development Unit National Center for Educational Achievement Michigan Technical Advisory Committee MDE/BAA Measurement Research & Psychometrics Unit Contributors
Identify a score for Proficient on the MME that represents being on track for career and college • Identify a score for Proficient on the highest grade level of MEAP that represents being on track to high school success • Identify a score for Proficient on the lower grade levels of MEAP that represent being on track to success in the next higher grade • Identify a score in each grade level of MEAP or MME that represents attainment of Partial Proficiency • Identify a score in each grade level of MEAP or MME that represents attainment of Advanced skills Issues to Address
Assumed that if a student is on track to success in college, then the student is also on track to success in technical career training • Therefore, focus on success in college • Included 2-year college programs (including job training programs at 2-year institutions) • Success could mean many things: • A, B, or C in college? • In a 2-year or 4-year college? • Conducted analyses of all of these scenarios • Conducted analyses only of academic success, not of every factor that leads students to be successful in college Defining On Track for Career and College
2-year versus 4-year colleges • Separate analyses were run regarding college success in 2-year and 4-year institutions • The cut scores identified for 2 year versus 4 year institutions were within measurement error of each other • Therefore, all final analyses combined all students from 2-year and 4-year colleges into a single group Defining College Success
Separate analyses were run using students achieving an A versus B versus C in their first credit-bearing freshman courses • A and C analyses did not produce usable results • Therefore, all final analyses used the criterion of B or better as the measure of college success • This is also the criterion for success used by ACT in its college readiness benchmarking study Defining College Success
Three types of analyses conducted • Logistic Regression (LR) • Signal Detection Theory (SDT) • Equipercentile Cohort Matching (ECM) Types of Analyses Conducted
Started with data from Michigan Public Institutions of Higher Education • Identified appropriate credit-bearing freshman courses against which to analyze the relationship between MME scores and course grades • Thanks to 2- and 4-year institutions for providing the data • Thanks to President’s Council for providing listings of courses appropriate to tie to MME scores • Using those data, conducted SDT analyses to connect MME to college freshman grades Identifying Proficient (On Track)Cut Scores on MME
Used SDT to map backward from 11th grade MME to 7th and 8th grade MEAP • Identified the score on MEAP that would maximize the consistent classifications from MEAP to MME • Used SDT to map from 7th grade MEAP to all other MEAP grades • Identified the score on each MEAP grade that would maximize consistent classifications from grade to grade Identifying Proficient (On Track) Cut Scores on MEAP
Recommend retaining the labels for the four performance categories • Not Proficient (Considered “Off Track”) • Partially Proficient (Considered “Off Track”) • Proficient (Considered “On Track”) • Advanced (Considered “On Track”) Achievement Levels
SDT can also identify scores on the MME scale that give certain probabilities of obtaining a B or better in the first credit bearing freshman course • Identified two probabilities that have strong meaning and give cut scores sufficiently far from the MME Proficient (On Track)cut scores. • 1/3 probability of B or better (Partially Proficient cut score) • 2/3 probability of B or better (Advanced cut score) • These cut scores were identified using SDT for the MME • These cut scores were also mapped back to the MEAP using ECM Identifying the Other Cut Scores
Impact of Recommended Cut Scores for MEAP and MME Mathematics (From Spring 2011 MME and Fall 2010 MEAP Data)
Mathematics Passing Rates (with Current and Recommended Cut Scores)
Approximate Percent Correct Scores needed to pass the Fall 2010 MEAP and Spring 2011 MME Mathematics tests with Current and Recommended Cut Scores
Impact of Recommended Cut Scores for MEAP and MME Reading (From Spring 2011 MME and Fall 2010 MEAP Data)
Reading Passing Rates (with Current and Recommended Cut Scores)
Approximate Percent Correct Scores needed to pass the Fall 2010 MEAP and Spring 2011 MME Reading tests with Current and Recommended Cut Scores
Impact of Recommended Cut Scores for MEAP and MME Science (From Spring 2011 MME and Fall 2010 MEAP Data)
Science Passing Rates (with Current and Recommended Cut Scores)
Approximate Percent Correct Scores needed to pass the Fall 2010 MEAP and Spring 2011 MME Science tests with Current and Recommended Cut Scores
Impact of Recommended Cut Scores for MEAP and MME Social Studies (From Spring 2011 MME and Fall 2010 MEAP Data)
Social Studies Passing Rates (with Current and Recommended Cut Scores)
Approximate Percent Correct Scores needed to pass the Fall 2010 MEAP and Spring 2011 MME Social Studies tests with Current and Recommended Cut Scores
MDE staff will apply the cut scores retroactively as if the new cut scores had been in place • Applied to four years of data • Within one month of approval • No retroactive accountability implications for schools, just to allow for following trends over time Reporting Implications
Joseph Martineau • Executive Director • Bureau of Assessment & Accountability • Michigan Department of Education • martineauj@michigan.gov Contact Information