RESEARCH nEWS: Standardized Testing: a Case study

1. RESEARCH nEWS:Standardized Testing: a Case study Tim Higgins April 15, 2013

2. Citation: • Feinman, J. (2008). High Stakes, but Low Validity? A Case Study of Standardized Tests and Admissions into New York City Specialized High Schools. Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/publication/high-stakes-but-low-validity

3. Great Quote • There are lies, damned lies and statistics. Mark Twain (attributed to)

4. The Specialized High School Admission Test (SHSAT) • State law (1971) – Admission to specialized public high schools based solely on this test given yearly. • Pool of applicants large, over 25,000 yearly. • Approximately 18% of all test takers were offered a seat at one of the schools with only half admitted to their first choice school. • Test consists of 100 items split between verbal and math • Four versions are given randomly.

5. The Issues • No study has ever been done that: • Compares SHS admissions procedure with generally accepted psychometric standards and practice. • Examines the statistical properties of the SHSAT itself: distribution of scores, test reliability, confidence intervals around scores, etc. • This study sought to remedy these deficiencies. • Mirrors national debate on standardized testing: • Critics contend that these tests are narrow measures of students’ abilities. • Difficult to distinguish among candidates with similar abilities and biased along racial, class, and gender lines.

6. Key Findings • Highly regarded and selective NYC public high schools base their admission on only this one test – the SHSAT. • Thousands of students are being denied admission whose scores are statistically indistinguishable from thousands who are admitted. • Unusual scoring feature: high score in one section and poor score in other yields better chance. • The scoring system may not be eliminating differences among the test versions. • This test may contain prediction bias across gender and ethnic groups. • Predictive validity studies are needed to link SHSAT scores to outcomes.

7. Admissions – Data analysis • Tables and Figures: • A. Table 1 – Summary Statistics, Figure 1:Frequency Distribution – Verbal Raw Scores (2006) • B. Table 2 – Summary Statistics, Figure 2:Frequency Distribution – Math Raw Scores (2006) • C. Grading the Test: Conversion • D. Figure 7: Balanced vs Unbalanced Scorers near cutoff (2006, Stuyvesant only) • Table 4 – Example of missed cut (2006, Stuyvesant only) • Figure 11: SHSAT Results by gender • Figure 12: School Enrollment by Ethnicity

10. Grading the Test: Conversion to Scaled Scores

11. Favoring unbalanced scores

13. Potential Bias

14. In the news New York Times 3/23/13 Education News 10/1/12

15. Recommendations • Formal predictive validity studies need to be carried out. At a minimum, these studies should look at the ability of SHSAT scores (separate verbal and math) and middle school grades to predict high school performance. • They should also test for prediction bias across gender and ethnic groups. • If this study concludes that it is best to use additional admissions criteria besides a standardized test, the New York State law—which says that admissions to these schools must be based solely on a test—would need to be changed.

16. Recommendations continued…. • Findings such as those presented in this study, and the particular choices of admissions procedures for these schools, should be discussed and deliberated in New York. • An informed decision should be made about future practices. Whatever admissions procedures are established, all applicants should know all of their implications. • These findings should also contribute to the broader national debate on standardized tests, school admissions, and high-stakes testing such as exit exams.

17. Implications – Social Justice

18. Discussion Questions 1) What could it be about the SHSAT that yields results contrary to most other studies about gender-based intelligences? 2) Is it simply a very difficult test that accomplishes a very competitive mission?