1 / 13

Gary W. Phillips Chief Scientist American Institutes for Research May 30, 2008

Obtaining International Benchmarks for States Through Statistical Linking: Presentation at the Institute of Education Sciences (IES) National Center for Education Statistics (NCES). Gary W. Phillips Chief Scientist American Institutes for Research May 30, 2008.

Albert_Lan
Download Presentation

Gary W. Phillips Chief Scientist American Institutes for Research May 30, 2008

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Obtaining International Benchmarks for States Through Statistical Linking:Presentation at the Institute of Education Sciences (IES)National Center for Education Statistics (NCES) Gary W. Phillips Chief Scientist American Institutes for Research May 30, 2008 This paper is intended to promote the exchange of ideas among researchers and policy makers.  The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.

  2. Two ways of obtaining International Benchmarks for States • Hard Way • Each State is administered PISA, TIMSS or PIRLS, and each country is administered PISA, TIMSS or PIRLS. • Very Expensive and impractical. • Easy Way (statistical linking) • Each State Is administered NAEP, and each country is administered TIMSS or PIRLS. • Inexpensive and practical. Dr. Gary W. Phillips American Institutes for Research

  3. PISA Cannot be Statistically Linked to NAEP • PISA is an age-based assessment (with grade 10 as the modal grade), NAEP is grade-based (grade 8) so PISA results cannot be statistically linked to NAEP. • Even if NAEP conducted a special study to make it possible to link NAEP and PISA (e.g., an age-based special assessment of NAEP with a modal grade of 10), State-NAEP still cannot be compared to PISA because State-NAEP is still in the 8th grade. Dr. Gary W. Phillips American Institutes for Research

  4. TIMSS was Purposefully Designed to be Statistically Linkable to NAEP • Same grades as NAEP • Similar content standards to NAEP • Similar test design • Matrix sampling of cognitive test items • Policy/research related Background questions • Similar nationally representative sampling • Similar scaling (item-response theory) • Similar analysis methods (plausible values) Dr. Gary W. Phillips American Institutes for Research

  5. What is Statistical Linking? • If you link test X to Y, this means • We are expressing the scores on one test (X) in terms of the metric of another test (Y) • For example, we are expressing the scores on NAEP (X) in terms of the metric of TIMSS (Y) • Four types of linking: equating, calibration, projection, moderation. Dr. Gary W. Phillips American Institutes for Research

  6. Assumptions in Linking Dr. Gary W. Phillips American Institutes for Research

  7. Why is it Important to Statistically Link NAEP to TIMSS or PIRLS? • Allows policy makers to see how the U.S. (as a whole), the states and school districts stack up against the rest of the world. • Provides a common metric that is familiar to US policy makers. It’s like converting world currencies to dollars. • Seeing the results of TIMSS or PIRLS through the lens of NAEP Achievement Levels provides a familiar benchmark to interpret international educational performance Dr. Gary W. Phillips American Institutes for Research

  8. What are the Ideal Data Requirements for Statistically Linking NAEP to International Assessments? • NAEP and the international assessment must be administered within the United States to groups of students which are: • Randomly equivalent • Nationally representative • In the same grades • In the same year • NAEP and the international assessment must cover similar (but not identical) content Dr. Gary W. Phillips American Institutes for Research

  9. Possible Statistical Linkages With NAEP During This Decade Dr. Gary W. Phillips American Institutes for Research

  10. Using NAEP linked to TIMSS as an International Benchmark in Mathematics(Phillips, “Chance Favors the Prepared Mind”, AIR, 2007) Dr. Gary W. Phillips American Institutes for Research

  11. Using NAEP linked to TIMSS as an International Benchmark in Science(Phillips, “Chance Favors the Prepared Mind”, AIR, 2007) Dr. Gary W. Phillips American Institutes for Research

  12. One potential way of Obtaining State Estimates for PISA is Through Small Area Estimation • If a survey has been carried out for the nation as a whole, the sample size may be too small for each state to generate accurate state estimates from the collected data. • To deal with this problem, it may be possible to supplement the survey data with auxiliary data (such as the CCD) and use regression modeling techniques in order to obtain state estimates. Dr. Gary W. Phillips American Institutes for Research

  13. Statistical Linking Versus Small Area Estimation • Statistical Linking - Two assessments (e.g., NAEP and TIMSS), measuring similar content, are administered to randomly equivalent national samples and the scales are statistically linked (sort of like converting Celsius to Fahrenheit). • Small Area Estimation – One assessment (e.g., PISA) is administered to a national sample and auxiliary data (such as the CCD) is used with regression modeling techniques to obtain state estimates. Dr. Gary W. Phillips American Institutes for Research

More Related