Educational Statistics: Avoiding Population and Sampling Pitfalls Meredith Wachs

1. Educational Statistics: Avoiding Population and Sampling PitfallsMeredith Wachs

2. Demographics of Johnson County, Tennessee 2008 Census Bureau Estimates: • Population: 18,112 (3.5% growth from 2000) • Ethnicities: 96% White, 3% Black, 1% Hispanic • High school graduates, age 25+: 58.4% (as of 2000, compared with 75.9% statewide) • Bachelor’s degree or higher, age 25+: 6.9% (as of 2000, compared with 19.6% statewide) • Mean travel time to work: 32 minutes (as of 2000)

3. More Demographics • Median Household Income (as of 2007): $30,447 (compared with $42,389 statewide) • Below poverty line: 21.9% (as of 2007, 15.8% statewide) • All these stats can be found at http://quickfacts.census.gov/qfd/states/47/47091.html • Mountain City, the county seat, had 2,383 residents as of 2000 (http://en.wikipedia.org/wiki/Mountain_City,_Tennessee).

4. Demographics continued… • Johnson County also has had a maximum security prison, Northeast Correctional Complex, which houses 1,819 inmates, since 1991 (http://www.tennessee.gov/correction/institutions/necx.html). • Unemployment as of February 2009: 11.8%; 9.5% statewide • As of per capita income of $19,680, Johnson County is the 4th poorest county in Tennessee (http://www.tennessee.gov/tacir/County_Profile/johnson_profile.htm).

5. Johnson County School System • Johnson County Schools include five elementary schools (pre-K-6), one middle school (grades 7-8), and one high school, along with an Extended Services Center, Headstart, and Adult Education programs. • They served 2,426 students in the 2005-2006 school year, with a 15:1 student-teacher ratio (NCES, http://www.education.com/schoolfinder/us/tennessee/district/johnson-county-school-district/). • Dropout Rate: 14.6% in 2007-2008 school year (10.1% statewide; http://www.tennessee.gov/tacir/County_Profile/johnson_profile.htm).

6. JC Schools continued… • In the 2005-2006 school year, Johnson County had 73.1% of students qualify for free or reduced lunch (rank 143 out of 155 systems in TN; state average: 52.9%). • 18.3% of students had disabilities (state average, 15.9%). • $7,973 were spent per student (rank 21; state average: $7,469) • Tennessee ranks 45th in spending. All data from Tennessee Education Association: http://www.teateachers.org/cms/Per+Pupil+Expenditures/103.html

8. No Child Left Behind • Nationwide, schools must test every year from 3rd-8th grade, and at least once in 10th-12th. • Originally just reading, writing, and math, schools must now assess science, too. • Highly qualified teachers must teach core subjects. • 95% of students must be present for testing, or three-year average must be 95%. • AYP (Adequate Yearly Progress) must be made in the state testing, along with graduation rate for secondary and (usually) attendance rate for elementary schools, for the whole school and each disability, economic, and racial subgroup. • http://www.stateline.org/live/ViewPage.action?siteNodeId=136&languageId=1&contentId=41611 • http://www2.edtrust.org/NR/rdonlyres/37B8652D-84F4-4FA1-AA8D-319EAD5A6D89/0/ABCAYP.PDF

9. NCLB continued • “No Child Left Behind, President Bush’s signature education reform law signed in 2002, is designed to raise academic achievement for all students and close gaps that separate students of color and low-income students from their peers by the 2013-2014 school year. All public schools and districts are subject to NCLB goals and reporting requirements. Schools or districts receiving Title I federal funds, which are earmarked for needy students and go to about 90 percent of public school districts in the U.S., are subject to penalties designed to close achievement gaps.” (Peterson, http://www.stateline.org/live/ViewPage.action?siteNodeId=136&languageId=1&contentId=41611).

10. NCLB continued… • “To account for those students with serious cognitive disabilities for whom grade-level standards are not appropriate, school districts and states can exempt up to 1% of all students… from grade-level testing.” • “Safe Harbor” provision: If a school system cuts the percent of non-proficient students by 10%, they are not penalized for not making AYP. • Schools are then assigned a “grade” for each subject and grade level; to earn an “A”, students may have to gain more than 1.5 school years in one year. • The overall goal is for 100% of students to be proficient in the tested subjects by 2014. • http://www2.edtrust.org/NR/rdonlyres/37B8652D-84F4-4FA1-AA8D-319EAD5A6D89/0/ABCAYP.PDF

11. What Happens When AYP Isn’t Reached? • Penalty timeline for schools "in need of improvement": • After 2 years: Schools must allow parents to transfer their children to other public schools. They also must develop a school improvement plan and spend 10 percent of their Title I allocation on teacher professional development. • After 3 years: Schools must provide eligible students with "supplemental services," which generally means tutoring.  • After 4 years: Schools must take "corrective action," such as replacing school staff, adopting new curriculum or extending the school day. • After 5 years: Schools face takeover by the state or a contracted private education firm. Taken from Peterson at http://www.stateline.org/live/ViewPage.action?siteNodeId=136&languageId=1&contentId=41611

12. How Tennessee Measures AYP • TCAP and writing tests are taken in elementary and middle school. • Three Gateway tests are required to graduate: Algebra I, Biology I, and English II. There is also an 11th grade writing assessment. • The number of items correct to pass the Algebra I test is 30 (out of 62). Advanced students have answered 42 correctly. • To be proficient on the English II Gateway, 26 questions must be answered correctly (out of 62). To be advanced, 39 must be correct. • Some additional end-of-course tests are taken for statistical purposes only. • Graduation rate is key for high school AYP; the TN long-term goal is 90% (http://www.all4ed.org/files/Tennessee_grp.pdf).

13. The Catch-It’s State-by-State! • In order to reach AYP (and to reach 100% proficiency by 2014), some states are being tougher than others. • “This has produced a dizzying array of incomprehensible results. In the first estimate of the law’s impact, Michigan was found to be home to 1,500 failing schools, while Arkansas had none. Most people didn’t think this made any sense in terms of those two states’ usual performances on tests.” (Gerald W. Bracey, Setting the Record Straight: Responses to Misconceptions About Public Education in the U.S., p. 72) • What should be done about this, if anything?

14. And so we have national tests… http://docs.google.com/gview?a=v&q=cache:vD1-0QrcweAJ:phs.prs.k12.nj.us/guidance/SAT_ACT_Comparison.pdf+SAT+vs.+ACT+scores&hl=en&gl=us&sig=AFQjCNEpfwrxk1aq0-RTru0Bvc2oKoXetA

15. Other Differences • According to an article in the New York Times, “the ACT is curriculum-based, while the SAT is aimed more at general reasoning and problem-solving skills.” • “The ACT lasts two hours, 55 minutes (plus 30 minutes with the optional writing test). The SAT lasts three hours, 45 minutes.” • “‘The bright underachievers who are bored and get through school using one quarter of their brains will do better on the SAT, because you just need good reasoning skills for that,’ says Scott White [a guidance counselor]. ‘And the overachievers, I don’t want to call them grinds, but they’re the ones who get the highest grades in the toughest classes because they work really hard, will do better on the ACT.’” (http://www.nytimes.com/2007/11/04/education/edlife/guidance.html).

16. Why Are ACT Scores Important (Besides Getting into College)? • “As part of our empirical definition of college readiness, ACT has established its College Readiness Benchmarks, which are the typical ACT scores that are used to place students into particular entry-level college courses across the United States: English Composition, College Algebra, introductory social science courses (such as Economics and U.S. History), and Biology. A student who meets a particular Benchmark has a 50 percent chance of earning a grade of B or above, or a 75 percent chance of earning a grade of C or above, in the corresponding college course. Students who take the ACT can therefore interpret their scores relative to these Benchmarks as another indicator of their readiness for college.” • -www.act.org

17. SAT vs. ACT By State Blue = SAT Red = ACT So we know that North Carolina spends more money on education then Mississippi (and is home to many fine higher education institutions!). Why, then, was Mississippi’s average SAT score higher than North Carolina’s in the early 90’s?

18. Know Your Population! • Socioeconomic status and background are very important to consider when looking at test scores. • Similarly, “The test makers’ statistics also indicate that members of minority groups score better across the board on the SAT than on the ACT. But that can be explained… Top students in all ethnic groups tend to take the SAT, while some Midwestern states require all juniors to take the ACT, thus lowering the mean.” (Slatalla, http://www.nytimes.com/2007/11/04/education/edlife/guidance.html). • Why would more top achievers take the SAT?

19. Another Example • The SAT had not been “recentered” since its average raw score was set at 500 in 1941. 500 was set by “10,654 students mostly living in New England and New York. Ninety-eight percent of them were white, 61 percent were male, and 41 percent had attended private college-preparatory high schools.” • Why was there a 20-year decline in the average score after 1963? • The SAT was eventually “recentered” in 1995 with scores of MANY more test takers! • This is featured in Bracey’s Setting the Record Straight.

20. A Few Definitions • Population- The entire group of individuals about which we want information • Sample- A part of the population from which we actually collect information, used to draw conclusions about the whole • Convenience sample- Sample selected by taking the members of the population that are easiest to reach; ex. mall surveys • A simple random sample of size n consists of n individuals from the population chosen in such a way that every set of n individuals has an equal chance to be the sample actually selected.

21. More Definitions • Outlier-An individual value that falls outside the overall pattern • Variable-Any characteristic of an individual. A variable can take different values for different individuals. • Confounded variables- Variables whose effects on the outcome cannot be distinguished from each other • What are some confounded variables possible for an outcome of low test scores? *All definitions taken from COMAP.

22. ACT, PLAN, and Explore • We have two populations: one that has taken the Explore and the PLAN, another who has taken the PLAN and ACT. • The tests are designed so 75% of test takers fall into the interval given by the previous test. What should this look like graphically? • How much do students have to improve to fall within their predicted range? • How should we sample these populations to get true representations of them?

23. Reality Hits! • For some reason, students with disabilities were underrepresented. • Many students took only one or the other, not both. There was also a LOT of missing data. • Many students had already taken the ACT and so did not retake on the school test day. What kind of scores would we have expected of them?

24. What do we do now? • We look at trimmed means (taking 10%, for example, from both ends of the data) or the median instead of the mean. Which one is better? • This gets rid of some of our outliers, but are our outliers significant? Let’s take a look at the data…

25. Explore Average Scores

26. Explore Benchmarks

27. PLAN Average Scores

28. PLAN Benchmarks

29. PLAN Results from Explore Predictions

30. Explore vs. PLAN Average Percentiles

31. ACT Average Scores

32. ACT Results from PLAN Predictions

33. ACT vs. Gateway

34. Explore, PLAN, and ACT Average Percentiles* * Not the same students tested on all three tests

35. ACT Benchmarks

36. Takeaway Points • ACT, not Gateway, standards and testing will tell us more about instruction and better prepare students for the future (especially post-secondary education), while fulfilling the Gateway standards. • This data is only helpful if it is: 1. Recorded 2. Analyzed every year 3. Reflective of true efforts by every student • How do we motivate non-college-bound students? • What else can we do with this kind of data (if sampled better)?