Best Practices in Quality Test Administration

1. Best Practices in Quality Test Administration Lessons Learned in the Administration of the National Assessment of Educational Progress (NAEP) Gina Broxterman, National Center for Education Statistics (NCES) June 20, 2016

2. NAEP Quality Assurance (QA): Purpose

3. NAEP QA: Pre-Administration • Item development • Items go through a multi-step quality assurance (QA) process before they are used in the operational assessment • Training of field staff • Field Staff are required to complete a combination ofdistance and in-person training • Transition to digitally based assessments • Configuration of equipment • Safeguards of delivery mode • Safeguards for system challenges

4. NAEP Assessment Administration

5. NAEP QA: Administration Best Practices NAEP Activities • Series of quality assurance debriefs • Item Development • Quality assurance reviews of items • Design, Analysis and Reporting • Review of performance data • Sample and Data Collection • NAEP field staff fill out debriefing questionnaires • Contractors host an in-person debrief meeting with all stakeholders to discuss all facets of NAEP 2016 administration • From the observation data, Sampling and Design contractor develops action items that will improve implementation for upcoming administrations

6. NAEP QA: Administration Best Practices NAEP State Coordinators (NSC) • Conduct observations • Provide feedback on the uniformity of the administration across all states • Embed QA practices in NSC data collection processes • Pre- and post-demographic data reviews • Convene working groups to revise processes for NAEP administration • Identified Issue: 2015 MyNAEP system was cumbersome for users • Action Plan: reduce jargon; reduce number of pages; add worksheets to streamline school's data collection; and provide multiple options for updating student lists

7. NAEP QA: Administration Best Practices HumRRO • Facilitate the Quality Assurance Technical Panel (QATP) to provide technical advice to NCES about potentially problematic NAEP issues, and design and conduct quality assurance research studies • Review NAEP contractor quality control plans and provide recommendations for improvements to NAEP operations and processes

8. Lessons Learned

9. NAEP: Lessons Learned - Translation Issue: Development schedule didn’t allow for thorough translation process Action Plan: Conduct translations earlier in the item development process Issue: Inefficient process for training NAEP scorers Action Plan: Develop an annotated guide that specifies purpose, style, format, and expected level of details, including the extent to which student responses should be quoted

10. NAEP 2016: Lessons Learned - DBA Issue: Session script for DBA assessments was not developmentally appropriate for fourth grade students Issue: Standardization of software and updates across large volume of equipment created potential risk. Action Plan: Associate the tablet's login screen background color with a particular software version so that field staff can easily identify the version type Action Plan: Develop grade-specific session scripts that provides additional guidance for fourth-grade students Issue: Tablets and the Pelican case were hard to transport/lift Action Plan: Create a best practices guide to be added to the field staff manual of best ways to transport the pelican case in various school settings

11. NAEP 2016: Lessons Learned - Training Issue: Creation of materials at the field staff level that opens potential for quality assurance issues. Action Plan: Develop additional quick check fact sheets for NAEP 2017 and organize in a central location for field staff Issue: Need more hands-on practice during in-person trainings Action Plan: Incorporate more hands-on practice with the field equipment

12. Future NAEP Administration:Quality Assurance

13. Future NAEP Administrations: QA Goals

14. Future NAEP Administrations: QA Goals