NAEP 12 th Grade Incentive Study

1. NAEP 12th Grade Incentive Study Henry Braun Boston College Irwin Kirsch Educational Testing Service PDII Conference Amelia Island February 8, 2007

2. Large-scale Survey Assessments are playing an increasingly important role in education policy • National Assessment of Educational Progress (NAEP) • National Adult Literacy Survey (NALS) and National Assessment of Adult Literacy (NAAL) • Trends in International Math and Science Studies (TIMSS) • Programme in International Student Assessment (PISA) • Progress in International Reading Literacy Study (PIRLS) • International Adult Literacy Survey (IALS) • Adult Literacy & Life Skills Survey (ALLS)

3. NAEP: A Brief History 1969 – Start of first national assessment (ECS as grantee) 1983 – ETS wins grant with a promise to incorporate “modern” statistical and psychometric methodologies 1990 – Introduction of Trial State Assessment 2002 – No Child Left Behind formally “embraces” NAEP for monitoring results (4th and 8th grades) for math and reading

4. NAEP Distinctions • Model for later large-scale education surveys • Utilizing leading edge psychometrics • The only national survey that permits comparisons across jurisdictions at one time point and within jurisdictions over time (cohort-to-cohort)

5. 12th Grade NAEP • National sample only • Not linked to NCLB • Heterogeneity in preparation of target population • Ongoing concerns about data quality • Object of renewed attention

6. Data Quality Concerns • NAEP is a no-stakes test administered to high school seniors in their last semester • School and student participation rates are lower than desirable • Assessed students may not be motivated to “try their best” • There may be systematic differences in motivation by reporting subgroups which are confounded with differences in targeted cognitive abilities.

7. Prospects for 12th Grade NAEP • Shift in focus from “achievement” to “readiness” • Expansion to state-level reporting • Link to current initiatives related to standards for high school diplomas.

8. Is it cost effective to invest $X millions in a major expansion of 12th Grade NAEP? This study is an opportunity to study how some aspects of data quality affect utility of 12th Grade NAEP with respect to: • Relevance • Comparability • Interpretability

9. Research Questions • Do 12th grade students taking the NAEP reading assessment and offered “strong” performance incentives display greater levels of engagement and/or achieve higher scaled scores on average than comparable students who are not offered such incentives? • Are there engagement or performance differences by treatment condition among students classified by ability level, by gender or by race/ethnicity? • If there are differences in performance by treatment group, what is the likely impact on the statistics reported by NAEP, as well as other indicators that are constructed from NAEP data? • Are there detectable differences between the control condition (fall administration) and the standard NAEP spring administration?

10. Effect of Motivation on Performance on Low-stakes Tests • O’Neill et al. (1992) [NAEP math in 8th and 12th grades] • NAGB Commission Report • Hoffman (2004) • Borghans et al. (2006) [College students tested on IQ-like items] • Segal (2006) [College students administered a coding test] • Conversations with George Loewenstein et al.

11. Study Methodology • Overview • Instrumentation • Incentives • Sample design and selection • Implementation • Statistical analysis

12. Methodology: Overview • Large enough sample to achieve reasonable power with respect to research questions [Power is affected by clustering of students within schools.] • Place results directly on the NAEP scale • Investigate strong financial incentives • Will not produce national norms • Fall administration

13. Methodology: Instrumentation • NAEP items are organized in blocks. Each block consists of a reading passage with a set of associated questions • The questions comprise both multiple choice and constructed response formats (short and extended answers) • There will be four blocks selected from released NAEP blocks. [Two blocks for “Reading for literacy experience” and two blocks for “Reading for information”.]

14. Methodology: Instrumentation (continued) • Each student is administered a booklet consisting of two blocks • There will be eight booklets randomly assigned to the student pool • Students will also fill out the NAEP student questionnaire • School administrators will complete the NAEP school questionnaire • Other?

15. Methodology: Instrumentation (continued)

16. NAEP Framework • Contexts for Reading • Reading for Information • Reading for Literary Experience • Aspects of Reading • Forming a General Understanding • Developing an Interpretation • Making Reader/Text Connections • Examining Content and Structure

17. Reading Passages for Study Reading for Literary Experience The Death of the Hired Man Robert Frost poem 1500 words, 175 lines The Flying Machine Ray Bradbury short story 1600 words

18. Reading Passages for Study Reading for Information Address to the Broadcasting Industry Excerpt from Newton Minnow’s 1961 speech to the National Association of Broadcasters 1500 words The Civil War in the United States: The Battle of Shiloh An entry from a Union soldier’s journal and an encyclopedia article about the battle 1900 words

19. NAEP Distribution of Student Time By Aspects of Reading 50% Forming a General Understanding & Developing an Interpretation 15% Making Reader/Text Connections 35% Examining Content and Structure

20. Study Booklets

21. Methodology: Incentives • Three conditions: Control and two incentives • Control condition • Students will be given standard NAEP instructions. After finishing they will be given a $5 debit card

22. Methodology: Incentives (cont.) • Incentive 1: • Students will be given standard NAEP instructions and told that at the conclusion of the session they will receive a debit card valued at $20 in appreciation for their participation and applying their best efforts to answer each item. They will also be asked to indicate which of two debit cards they would like to have. The cards will be linked to different stores. It is hoped that the effect of the incentive will be enhanced by having the students actively make a choice in advance of the assessment.

23. Methodology: Incentives (cont.) • Incentive 2: • Students will be given standard NAEP instructions and told that at the conclusion of the session they will receive a debit card valued at $5. In addition, two questions will be selected at random from the booklet. The debit card will be increased by $10 for each correct answer, so they can receive a maximum of $25. They will also be asked to indicate which of two debit cards they would like to have. The cards will be linked to different stores. It is hoped that the effect of the incentive will be enhanced by having the students actively make a choice in advance of the assessment.

24. Methodology: Sample Design • A supplemental school sample to the national school sample for the Spring 2008 administration will be selected • Sixty schools in six states enrolling (in the aggregate) a diverse student population • Sixty students in each school will be selected at random—organized into three “classes” for administration

25. Methodology: Implementation • Recruit schools and students • Develop instructions for incentive conditions • Assemble, print and ship booklets • Train assessment administrators • Administer assessment and fulfill incentive obligations

26. Methodology: Implementation (cont.) • Carry out on-site QC and ship booklets • Process background data • Score and process cognitive data • Construct and transmit data file for analysis

27. Methodology: Statistical Analysis • Data editing • Comparisons to historical data • Scaling of data (very complex!) • Construction of NAEP score file • Statistical analyses to address each Research Question

28. Possible Outcomes (and their Interpretations) • Null effect • Mean effect (no interactions) • Mean effect (with interactions)

29. DISCUSSION • Can we improve structure and/or implementation of incentives (to maximize “treatment effect”)? • What are policy implications of the different scenarios? • Other suggestions to enhance the value of the study?