Design of the 2011 NAEP-TIMSS Linking Study

1. Design of the2011 NAEP-TIMSS Linking Study Andrew Kolstad Senior Technical Advisor to the Assessment Division National Center for Education Statistics

2. Goals of the study in 2011 For states and school districts, to project TIMSS mathematics and science scores from performance on NAEP (without the cost of participating in TIMSS)

3. Previous NAEP / TIMSS linkings - 1 • Johnson’s statistical linkage to 1996 state NAEP • No students took both NAEP and TIMSS assessments • TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical moderation” (matching score distributions) • Evaluated with TIMSS in Minnesota and Colorado 1995 • The actual grade 8 means for Minnesota fell within the confidence intervals • The actual grade 4 means for both Colorado and Minnesota were higher than the predicted means

4. Previous NAEP / TIMSS linkings - 2 • Johnson and Cohen’s projection linkage to 2000 grade 8 state NAEP mathematics and science • A linking sample of students was assessed by NAEP in the winter of 2000 and again by TIMSS in the spring of 2000 • TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical projection” (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores) • Evaluated with grade 8 TIMSS in twelve states in 1999 • The performance differed: the actual grade 8 means generally fell above the confidence intervals for the predicted means • No TIMSS score projections were published

5. Previous NAEP / TIMSS linkings - 3 • Phillip’s statistical moderationlinkage in 2007 • International grade 8 TIMSS-equivalent scores of NAEP achievement level cut scores projected using “statistical moderation” (no student took both assessments) • Based on national distributions of NAEP 2000 mathematics and science scores matched to U.S. score distributions on • TIMSS 1999 assessment • TIMSS 2003 assessment • Result: Estimated percentage of basic, proficient, and advanced students in each country that participated in the 1999 TIMSS and 2003 TIMSS studies

6. Improving on previous linking designs • How the current linking study differs • Both NAEP and TIMSS in 2011, the same year this time • Two linking samples of students; no student assessed in both winter and spring • Two booklet designs; one looks like NAEP in the winter and the other looks like TIMSS in the spring • Projected TIMSS mathematics and science scores predicted from NAEP state scores using “statistical projection” • (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores) • Evaluated with grade 8 TIMSS in eight states in 2011

7. Components in linking NAEP & TIMSS • NAEP state-level science and mathematics data needed as a basis for projecting TIMSS mathematics and science scores • For the first time in NAEP history, all 50 states volunteered to participate in a voluntary state NAEP • TIMSS state-level grade 8 data needed to check the validity of the linking function • Linking samples of students will provide responses to items from both assessments; needed to estimate relationships • No students will take NAEP in the winter and TIMSS in the spring, to avoid the decline in scores found in the previous study

8. Booklet designs for NAEP & TIMSS • NAEP booklets contain either science or mathematics, but not both • TIMSS blocks contain both science and mathematics blocks

9. TIMSS assessment booklets Each block appears in two booklets Math & Science Each of the 14 booklets includes two blocks of mathematics and two blocks of science items

10. Linking study approach • Administer braided booklets in both testing windows • NAEP in Winter (January to March, 2011) • TIMSS in Spring (April to June, 2011) • Select national samples independent of the main NAEP and TIMSS samples • 10,000 students in NAEP in Winter (half science, half math) • 7,500 students in TIMSS in Spring (half science, half math) • Collect the data using NAEP field staff in winter and TIMSS field staff in spring (Westat is the contractor for both) • Score the items using the same scoring staff with the same training and QC procedures that NAEP and TIMSS use

11. Booklet designs for braided study NAEP-window booklets contain either science or mathematics blocks, but not both • TIMSS-window booklets contain bothscience and mathematics TIMSS blocks • TIMSS-window booklets contain three blocks of science or math, plus one block of NAEP

12. Block composition of braided booklets Example booklet configurations in the TIMSS window TIMSS Science block TIMSS Math blocks NAEP Math block Mathematics TIMSS Science blocks TIMSS Math block NAEP Science block Science

13. TIMSS at the state level • Planned as a validity check on the NAEP projection methodology • Administered as part of regular, national TIMSS • Funded by NCES, not by states • 29 states volunteered; • 8 states will participate • MA, MN, AL, CA, CT, IN, CO, & NC

14. NAEP at the state level • A state-level grade 8 mathematics assessment was already planned • All states have volunteered to participate in the new 2011 NAEP science assessment • Science will not be a regular NAEP state-level assessment with full reporting (no urban districts) • The braided linking samples are national only, with a very small sample in any given state

15. Timeline for NAEP-TIMSS linking • Release of the national and state TIMSS results is scheduled for December 2012 • Linking results will be reported a few months after the main TIMSS release

16. Benefits of the proposed approach • Avoids interference with the conduct of regular operational assessments • Provides a NAEP-TIMSS link through common items taken by the same student • Measures the amount of growth during the period from winter to spring • Provides states with international benchmarks in science and mathematics at grade 8, without the cost of conducting TIMSS at the state level