The Language of Enrollment Data Statistics, Psychometrics, and Other Technical Matters

1. The Language of Enrollment Data Statistics, Psychometrics, and Other Technical Matters GACRAO 2006 John Albright College Board Southern Regional Office jalbright@collegeboard.org

2. “Without data, you’re just another person with an opinion.”

3. Just Another Dayon the Road Demographics Descriptive Statistics Percentiles Just Another Dayin the Office Standard Error of the Difference Group Differences and Fairness Predicting College Performance Outline

4. One Useful Resource The SAT Program Handbook • Updated every summer to counselors, admissions offices • The PDF can be downloaded from www.collegeboard.com/highered/ra/sat/sat_resources.html

5. Another Resource College Board Demographics Project • Publications • Workshops • Trends • PowerPoints • ”Higher Education Landscape” • www.collegeboard.com/highered/de/index.html

6. DEMOGRAPHICSPublic School Enrollment 1965−2014 Source: NCES. The Condition of Education 2005

7. Number of High School Graduates Total U.S. 1994−2018 by Race/Ethnicity Source: WICHE. Knocking at the College Door: Projections of High School Graduates by State, Income and Race/Ethnicity 1988-2018.

8. Public School EnrollmentPercent Change by Region 2005–2010 Source: NCES. Projections of Education Statistics to 2014 and 2013

9. Georgia Public High School Graduates 2005–2018 Source: WICHE. Knocking at the College Door: Projections of High School Graduates by State, Income and Race/Ethnicity 1988-2018.

10. Descriptive Statistics Help describe, organize, and summarize data so it can be better understood • Frequency Distributions • Measures of Central Tendency (Averages) • Range and Standard Deviation

11. Raw Data for Hypothetical U.

12. Frequency Distribution Number of Hypo U. Applicants at Each SAT Critical Reading Score Interval (N=1000) Note. Percentages may not add to 100% due to rounding.

13. Histogram Representing Critical Reading Scores of Applicants at Hypo U.

14. Measures of Central Tendency(Averages) • Mean • Median • Mode

15. Mean (Arithmetic Average) 710 530 600 520 410 490 510 490 • Consider the following set of 8 SAT math scores from the Hypo U. data. The sum is 4,260, so the mean is:

16. Median 710 600530 520 510 490 490 410 • The same set of SAT math scores has been arranged from highest to lowest. • One half of the these students scored above the median; the other half scored below.

17. Mode 710 600 530 520 510 490 490 410 • The score that appears most often. • For this set of SAT scores, there is one score that appears two times: 490. • Therefore, 490 is the mode.

18. Inter-Quartile Range(a.k.a. the “Middle Half”) The inter-quartile range is the range of scores between the 25th and 75th percentiles, or the middle half of a distribution.

19. 2006 College HandbookInter-Quartile Range

20. Percentiles • Compare an individual’s score to groups of other students who took the test • SAT percentiles: • National “College-Bound Seniors” • State SAT Takers • Other Groups of SAT Takers (see www.collegeboard.com/satdata) • Other Groups (i.e., High Schools and Colleges)

21. NationalPercentiles 2006 Percentiles www.collegeboard.com/satdata

22. NationalPercentiles SAT Program Handbook, 2005–2006, p. 24. 570 69 65

23. Sample SAT Report National and state percentiles are displayed on SAT score reports to provide additional information on the student’s performance.

24. Percentiles for 2005 SAT Math

25. Preparing for the SAT(Coaching) Probably not Useful • Short-term cramming or drilling • Learning “tricks” Useful • Test-taking familiarization • Instruction focused on understanding the knowledge and abilities in a domain (tutoring)

26. Information on coaching can be misleading • Individual examples are not realistic indications of the average effects of coaching or how another student may do as a result of coaching. • Score increases could be the result of many things.

27. Scores do change(2005 SAT Program Handbook, p.25)

28. Scores Do Change Critical Reading (Verbal) Junior Year Scores

29. Scores do change 530–570

30. Scores do change Percentage of Students with Senior Year Score Gain or Loss

31. -80 to-100 -50 to-70 -20 to-40 -10 to+10 +20 to+40 +50 to+70 +80 to+100 +110 to+130 2 7 18 28 25 14 4 1 Scores do change

32. Scores do dhange Average of Senior Year Score 559

33. Scores do change Average Score Gain or Loss 10

34. Why do scores change? • Real academic growth • Practice • Standard Error of Measurement

35. 1. Score change due to real academic growth During the course of an academic year, students are likely to learn and improve upon numerous reading, writing, and math skills tested on the SAT.

36. 2. Score change due to practice Although the “practice effect” may be more pronounced for students with very limited experience with standardized tests, even an experienced test taker will likely benefit somewhat from practice.

37. SAT Readiness Program Free Resources • SAT Preparation Booklet™ • SAT Preparation Center™ • The Official SAT Question of the Day™ Student Resources • The Official SAT Study Guide™ • The Official SAT Online Course™ • The Official SAT Question of the Day Box Calendar

38. 3. Score change due to theStandard Error of Measurement • No matter how well-constructed a test is, it cannot measure with perfect precision. • This imprecision is called the standard error of measurement (SEM).

39. SAT and the Standard Error of Measurement An SEM of 30 indicates that a student’s true score (only known if a student took a test with every possible question that could be asked) generally falls within a band of his/her obtained score, plus or minus 30 points.

40. Score rangesfor the SAT The SAT Score Report includes information about the SEM

41. Coaching research • On average, as a result of formal coaching: • verbal scores increase between 9 and 15 points • math scores increase between 15 and 18 points • Longer coaching programs result in slightly larger score gains than shorter programs. After approximately 3–9 hours, additional time devoted to formal test preparation appears to have diminishing returns.

42. When evaluating admission applicants, consider… • Standard Error of Measurement (SEM) • Used when looking at one student • About 30 (40) points • Standard Error of the Difference (SED) • Used when comparing two students • About 40 (50) points

43. When evaluating admission applicants, also consider… That the SAT is an excellent measurement of certain important aspects of college readiness, specifically: • Critical thinking, reasoning, and writing skills There are also other important skills, abilities, and personal qualities that many colleges are interested in, such as: • Creativity, community service, perseverance, etc.

44. Validity • Content Validity covers a representative sample of the subject domain to be measured • Face Validity “looks valid” to students taking it, administrators deciding on its use, and other technically untrained observers • Predictive Validity is effective in predicting an individual's behavior in specified situations

45. Predictors Test Scores HS Grades Class Rank Other Outcomes College Grades Faculty Endorsement Retention Success in Life Predicting College Performance:How well do different measures used in the admissions process predict important outcomes in college?

46. Scatter Plot of SAT-Math Scores and Freshman Year GPA This particular line represents a correlation (expressed as “r”) of .52.

47. Predictive Equation • A prediction equation is a formula that can be used to predict an applicant’s first-year grades in college based on their high school grades and SAT scores. FYGPA’ = a + b1(HSGPA) + b2(SAT-W) + b3(SAT-M) + b4(SAT-CR) Where FYGPA’ is the predicted value, a is the intercept, and b1 to b4 are the regression weights. • Once we analyze the data and estimate the weights, we can plug in an individual’s scores on each of the predictors and predict their FYGPA. 3.34 = .601 + .310(3.56) + .0011(680) + .0008(610) + .0007(570)

48. Correlations of HS GPA and SAT scores with first-year college GPA(SAT Program Handbook, p.29)

49. Validity Four-Year Degree Attainment Rates by High School Grades and SAT Scores N = 56,818 first-year students at 262 institutions

50. Group Differences • Gender • Racial/ethnic groups • English language learners (ESL, LEP) • Socio-Economic Status groups • Parental education • Family income • Other