1 / 26

A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan

Establishing a statistical linkage between the NAEP and ECLS-K grade 8 reading assessments using a common sample, and testing the validity of the link.

williemcdonald
Download Presentation

A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics Services Institute American Institutes for Research

  2. Purpose • In Spring of 2007, about 1300 studentstook the NAEP and ECLS-K grade eight reading assessments • The purpose of this study is to establish a statistical link between these two assessments using this common sample • We also test the validity of the link by comparing the model projected results to reported NAEP results

  3. Background

  4. Early Childhood Longitudinal Study Kindergarten Cohort (ECLS-K) • Provides longitudinal data on students’ academic achievement in reading, mathematics, and science • Began in the fall of 1998 with a national sample of 21,000 kindergartners • Each student is tested by a series of two-stage, 30-minute adaptive tests, administered face to face, on seven occasions • Item parameters and individual scores are estimated using IRT

  5. NAEP 8th Grade Reading 2007 • Provides cross-sectional data on students’ academic achievement in reading • Nationally representative sample of more than 350,000 students participated in the 2007 reading assessment • Each student takes just a portion of the test (BIB Design), consisting of two 25-minute sections or one 50-minute section • Item parameters are estimated using IRT, but results are reported at group level. NAEP does not report individual scores

  6. Differences between NAEP and ECLS-K • The two assessments have similar frameworks, but do not share any common items • NAEP uses a nationally representative sample • ECLS-K is representative of children enrolled in first grade in the U.S. in 1999-2000

  7. Estimating the projection equation

  8. Linking through a common sample • In Spring of 2007, about 1300 public school students took NAEP and ECLS-K grade 8 reading assessments. • About ¼ were part of the operational NAEP sample • The rest took the NAEP assessment for the purposes of the common sample linking study

  9. Method and challenges • NAEP scores in the form of plausible values were not available for about ¾ of the linking sample since they were not part of the operational NAEP assessment • Using item level data from the entire common sample, MML regression was used (with the AM software) to estimate a projection equation predicting NAEP score distributions from ECLS-K scores • The projection equation was • where y is the NAEP score • and x is the ECLS-K score

  10. The projection equation • The projection equation was estimated using two different weights: • the original ECLS-K sampling weights, and • poststratification weights Parameter estimates predicting NAEP scale scores from ECLS-K scale scores 1 Using original weights: RMSE = 18.84, F(1,168) = 326.47 2 Using poststratification weights: RMSE = 18.60, F(1,168) = 370.64

  11. Variance of the projection • For the projection equation, • the a, b parameter estimates are denoted • and their covariance as • The projected NAEP average reading score is • The variance of the projection at mean is

  12. Testing the validity of projection equation

  13. Comparing reported and projected NAEP mean National average scores and 95% confidence intervals for projected and reported 2007 NAEP reading scores: grade 8, public

  14. Scores by gender and race/ethnicity • Left bar: Confidence Interval for the mean based on the projection using the original weights • Right bar: Confidence Interval for the reported NAEP mean NAEP Scale

  15. AN APPLICATION OF THE LINK: Reading performance at earlier grades and proficiency in NAEP

  16. Early childhood reading performance and proficiency in NAEP What is the relationship between reading performance at first, third and fifth grades and Proficiency in eighth-grade NAEP reading assessment?

  17. Proficiency in eighth-grade NAEP reading assessment • Show an overall understanding of the text, including inferential as well as literal information • Extend the ideas in the text by making clear inferences from it, by drawing conclusions, and by making connections to their own experiences—including other reading experiences • Identify some of the devices authors use in composing text

  18. ECLS-K Reading performance levels • Level 1 : Letter recognition: identifying upper- and lower-case letters by name • Level 2 : Beginning sounds • Level 3 : Ending sounds • Level 4 : Sight words • Level 5 : Comprehension of words in context • Level 6 : Literal inference • Level 7 : Extrapolation • Level 8 : Evaluation • Level 9 : Evaluating nonfiction • Level 10: Evaluating complex syntax: evaluating complex syntax and understanding high-level nuanced vocabulary in biographical text.

  19. Reading performance at grade 5 and Proficiency in 8th grade NAEP • Percentage of students at Levels 5 through 9: Grade 5

  20. Reading performance at grade 5 and Proficiency in 8th grade NAEP • Percentage of students at Levels 5 through 9: Grade 5 48 % 33 % Evaluation: demonstrating understanding of author’s craft … 72 % 13 %

  21. Reading performance at grade 3 and Proficiency in 8th grade NAEP • Percentage of students at Levels 4 through 8: Grade 3

  22. Reading performance at grade 3 and Proficiency in 8th grade NAEP • Percentage of students at Levels 4 through 8: Grade 3 46 % 62 % 25 % Extrapolation: identifying clues used to make inferences, … 11 %

  23. Reading performance at grade 1 and Proficiency in 8th grade NAEP • Percentage of students at Levels 3 through 7: Grade 1

  24. Reading performance at grade 1 and Proficiency in 8th grade NAEP Percentage of students at Levels 3 through 7: Grade 1 49 % 34 % 73 % Comprehension of words in context: reading words in context 13 % 85 %

  25. Summary of findings • Using a common sample, a linking equation was estimated • Using the equation, projected NAEP scores were computed for ECLS-K 8th graders • Mean projected NAEP scores for the nation and the gender and racial/ethnic groups were close to reported NAEP results • Mean projected NAEP scores were used to examine the relationship between proficiency in grade eight NAEP reading and earlier reading performance

  26. Discussion • Limitations • Composition of the ECLS-K sample and the population it represents • Projection results for Hispanic students • Future analyses • Further validation of the link • Modeling growth in reading on the NAEP scale

More Related