NAEP 2011 School coordinator Workshop

1. NAEP 2011School coordinator Workshop Heather HeinekeNAEP CoordinatorOffice of Instruction, Assessment, & CurriculumRhode Island Department of EducationNovember 29 & December 9, 2010 1

2. The Goal for Today: 2 Goal: To provide you with the information you need to create a successful NAEP administration. Agenda: What is NAEP? The NAEP Process and Schedule Accommodations Including and Excluding Students (new policy) Pre-Assessment Visit

3. The Nation’s Reportcard What is NAEP? 3

4. What is NAEP ?National Assessment of Educational Progress 4 Funded by Congress Mandatory in reading and math in grades 4 & 8 Informal verification of state assessment results. State and national trend data NAEP State Coordinators in each state, Puerto Rico, the District of Columbia, the Department of Defense Schools, the Bureau of Indian Affairs, and some large school districts (TUDA) “Informal verification of statewide results” – this does not mean that the percentage of students on the NECAP must equal the number proficient on the NAEP assessment. What this means is that if the percentage of students at or above proficient on the NECAP goes up significantly, so should the NAEP percent of students at/above proficient. It is important to remember that state assessments exist for a number of different reasons: instructional support, alerts teachers and parents to potential areas of weakness and strength of a student, program evaluation (does summer school really work??), and accountability. NAEP doesn’t exist for those reasons. It’s only concern is to gauge how students are performing nationally and across states. Remember these differences don’t mean the NAEP is better or worse than the NECAP – they are different tests designed for different reasons. Both tests are valuable for the very reasons that set them apart.“Informal verification of statewide results” – this does not mean that the percentage of students on the NECAP must equal the number proficient on the NAEP assessment. What this means is that if the percentage of students at or above proficient on the NECAP goes up significantly, so should the NAEP percent of students at/above proficient. It is important to remember that state assessments exist for a number of different reasons: instructional support, alerts teachers and parents to potential areas of weakness and strength of a student, program evaluation (does summer school really work??), and accountability. NAEP doesn’t exist for those reasons. It’s only concern is to gauge how students are performing nationally and across states. Remember these differences don’t mean the NAEP is better or worse than the NECAP – they are different tests designed for different reasons. Both tests are valuable for the very reasons that set them apart.

5. The Facts: 2011 5 168 schools (grades 4 & 8) 3,000 grade 4 students and 3,000 grade 8 students 8 middle schools giving the computer-based Math and Writing assessments 90-minutes from start to finish. Does not include accommodation sessions or special studies like the computer-based tests. NO test materials for schools to keep track of. All materials will be handled by the NAEP field staff Your Pre-Assessment Visit (PAV) with the Assessment Coordinator (AC) assigned to your school is the key to a successful NAEP administration

6. NAEP Has Many Different Parts 6 The 2011 “Main” NAEP sample is comprised of two parts – National and State NAEP. No one except the sampling experts know which student booklet will be assigned to these studies. The TUDA districts are over-sampled in order to get enough data to create reliable district results. The TUDA program is steadily expanding and now includes 21 districts. These districts will receive NAEP results separate from their respective state results. It is unclear when NCES will feel it has enough data to open this district program up to even more states/districts or make it an official aspect of the NAEP assessment program. The Long Term Trend is a separate administration done in the even years that includes students of certain ages. The Long Term Trend started in the late ’60’s and has remained unchanged.The 2011 “Main” NAEP sample is comprised of two parts – National and State NAEP. No one except the sampling experts know which student booklet will be assigned to these studies. The TUDA districts are over-sampled in order to get enough data to create reliable district results. The TUDA program is steadily expanding and now includes 21 districts. These districts will receive NAEP results separate from their respective state results. It is unclear when NCES will feel it has enough data to open this district program up to even more states/districts or make it an official aspect of the NAEP assessment program. The Long Term Trend is a separate administration done in the even years that includes students of certain ages. The Long Term Trend started in the late ’60’s and has remained unchanged.

7. NAEP Field Staff Structure 7

8. The NAEP Test Design 8 Each student will take ONE CONTENT AREA TEST. Tests will be given at the same time in the same place. SPIRALED BOOKLETS (test forms are mixed up) COMMON & MATRIX ITEMS but no field test items in NAEP. Multiple choice Short answer Extended response HUMAN SCORING for the short answer and extended response, MACHINE SCORING for multiple choice.

9. NAEP Assessments for 2011 Reading Mathematics Writing Science Computer-based Writing Computer-based Mathematics 9

10. The NAEP Schedule NAEP Testing Window:  Jan. 24 – March 3rd, 2011 Students Lists Arrive: in PAV packets after Jan. 3rd. Pre-Assessment Visits (PAVs):  Jan. 3rd – Jan. 24th (AC’s will begin calling schools to make appointments in December)  10

11. The Process 11 Process for most schools…GO THROUGH PROCESS…However, some schools may have to submit their Exclusion Forms before they get their student lists in order to give me enough time to approve the exclusions. Remember: 4th grade is a take all and 8th grade is mostly take all…you actually don’t have to have your student list to do this. Process for most schools…GO THROUGH PROCESS…However, some schools may have to submit their Exclusion Forms before they get their student lists in order to give me enough time to approve the exclusions. Remember: 4th grade is a take all and 8th grade is mostly take all…you actually don’t have to have your student list to do this.

12. Accommodations 12

13. Accommodations: What you need to do 13

14. Accommodations on NAEP 14 These are the accommodations cross-walks that will be sent in your PAV packet that you will receive in January. There is one set for Students with Disabilities and one for ELLs. For SD/ELL students can have accommodations from both lists as necessary Special education/ELL teachers can help with the assessments. These are the accommodations cross-walks that will be sent in your PAV packet that you will receive in January. There is one set for Students with Disabilities and one for ELLs. For SD/ELL students can have accommodations from both lists as necessary Special education/ELL teachers can help with the assessments.

15. Accommodations to Combine 15 There aren’t many accommodations that are different from NECAP. Since we just finished NECAP, many of the accommodations you arranged for your students will be recreated for NAEP. If you made a list of accommodations for students taking NECAP, this should be relatively simple. There are only a couple of things to point out: how to read the chart and how extended time is administered. The AC assigned to your school will cover this all with you on the day of your PAV. However, if you have questions, please call me. There aren’t many accommodations that are different from NECAP. Since we just finished NECAP, many of the accommodations you arranged for your students will be recreated for NAEP. If you made a list of accommodations for students taking NECAP, this should be relatively simple. There are only a couple of things to point out: how to read the chart and how extended time is administered. The AC assigned to your school will cover this all with you on the day of your PAV. However, if you have questions, please call me.

16. Reading Directions Allowed In Spanish 16 Another accommodation to explain.Another accommodation to explain.

17. Approved Bilingual Dictionaries for ELL Students Bilingual dictionaries are allowed on the Math, Science, and Writing tests but not Reading. Download from: http://www.ride.ri.gov/assessment/NECAP.aspx NOTE: Make sure the student has time to use the dictionaryBEFORE the test. 17

18. New Monitoring Procedures & Guidelines Including Students with Disabilities & ELLs on NAEP 18

19. The New Policy: Rhode Island’s Inclusion and Exclusion Rates for NAEP NAEP needs to be as inclusive as possible Some states are not including students To get those exclusion rates down (and inclusion rates up), states that don’t meet requirements will be “flagged” in national reports. 19 NAGB (National Assessment Governing Board) is concerned that states are not including enough students in NAEP. Some states do exclude well above 10% of some populations of students. In response to these exclusion rates, NAGB instituted a new policy of flagging states who do not meet two new criteria. The first is an overall inclusion rate of 95%. It is very similar to our own “participation rate”. The second criteria is a participation rate of at least 85% of the students with disabilities and English language Learners in reading and math. NAEP provided each state with a exclusion rate data from 2009 so states could see if they had any problems in these areas.NAGB (National Assessment Governing Board) is concerned that states are not including enough students in NAEP. Some states do exclude well above 10% of some populations of students. In response to these exclusion rates, NAGB instituted a new policy of flagging states who do not meet two new criteria. The first is an overall inclusion rate of 95%. It is very similar to our own “participation rate”. The second criteria is a participation rate of at least 85% of the students with disabilities and English language Learners in reading and math. NAEP provided each state with a exclusion rate data from 2009 so states could see if they had any problems in these areas.

20. Including and Excluding Students:Mathematics 2009 Students with Disabilities English Language Learners 20 These are the Exclusion rates for 2009 for Mathematics for students with disabilities and ELLs for grades 4 & 8 . According to the 2009 Exclusion rates, Rhode Island would be flagged for excluding more than 15% of ELL students for 8th grade math.These are the Exclusion rates for 2009 for Mathematics for students with disabilities and ELLs for grades 4 & 8 . According to the 2009 Exclusion rates, Rhode Island would be flagged for excluding more than 15% of ELL students for 8th grade math.

21. Including and Excluding Students:Reading 2009 Students with Disabilities English Language Learners 21 These are the Exclusion rates for Reading students with disabilities and ELLs for grades 4 & 8. In Reading, Rhode Island would be flagged for not excluding more than 15% of the students with disabilities in both grades 4 and 8 as well as excluding ELL students in the 8th grade. So in collaboration with RI’s ELL and special education experts, we developed a new policy regarding excluding students. We hope that this new policy does two things: provides us with data about why students are excluded from NAEP, and helps us monitor the exclusion rates accurately. These are the Exclusion rates for Reading students with disabilities and ELLs for grades 4 & 8. In Reading, Rhode Island would be flagged for not excluding more than 15% of the students with disabilities in both grades 4 and 8 as well as excluding ELL students in the 8th grade. So in collaboration with RI’s ELL and special education experts, we developed a new policy regarding excluding students. We hope that this new policy does two things: provides us with data about why students are excluded from NAEP, and helps us monitor the exclusion rates accurately.

22. Reasons for Excluding Students Alternate Assessment (RIAA) Required accommodation not provided Homeschooled / outplacement Enrolled in a U.S. school for less than 12 months AND selected for Reading or Writing (with exception below) Takes the RIAA Homeschooled/ outplacement Students with Disabilities English Language Learners 22 These are the reasons students should be excluded from the NAEP assessment. To provide guidance on when to include ELL students new to the country, if a student scores 4.0 on the W-APT literacy composite score, that student should be allowed to participate on the NAEP.These are the reasons students should be excluded from the NAEP assessment. To provide guidance on when to include ELL students new to the country, if a student scores 4.0 on the W-APT literacy composite score, that student should be allowed to participate on the NAEP.

23. Two Parts to the New Monitoring Process MUST receive permission to exclude students. ONLINE reporting done by AC assigned to your school. 23 So, now that we covered the exclusion rates, there are two parts to the new monitoring process.So, now that we covered the exclusion rates, there are two parts to the new monitoring process.

24. 24 This is page 4 of your NAEP 2011 Accommodations policy that you will receive in your PAV packet. There is one for students with disabilities and one for ELL students. The requirement is that these must be faxed to me one week before your PAV. Schools with January testing dates may have problems with this deadline. The most important requirement is that I receive this before your PAV with enough time to approve your exclusions and discuss any issues with you. Because you are discussing accommodations and exclusions at your PAV, I need to approve these exemptions before that meeting.This is page 4 of your NAEP 2011 Accommodations policy that you will receive in your PAV packet. There is one for students with disabilities and one for ELL students. The requirement is that these must be faxed to me one week before your PAV. Schools with January testing dates may have problems with this deadline. The most important requirement is that I receive this before your PAV with enough time to approve your exclusions and discuss any issues with you. Because you are discussing accommodations and exclusions at your PAV, I need to approve these exemptions before that meeting.

25. 25 The differences here are the test the student was selected for and their W-APT score.The differences here are the test the student was selected for and their W-APT score.

26. 26 Monitoring Process Part 2: Online Reports This is the second part of the monitoring process and it is completed online by the AC. The AC, field manager, and I share a data base called the School Control System (SCS) and it has all of the information about when you are testing, how many sessions you have, who the testing coordinator is, etc., and now it has a new feature where I will be able to generate this list and compare it to the one you fax to me. This way, the process and the data are more valid. I want to point out that this new process is not to force you to include every student selected and to not be flagged – although not getting flagged would be nice – it is to justify each exclusion in case we do get flagged. We need to be able to show that students who were excluded should have been. This type of data should also be meaningful.This is the second part of the monitoring process and it is completed online by the AC. The AC, field manager, and I share a data base called the School Control System (SCS) and it has all of the information about when you are testing, how many sessions you have, who the testing coordinator is, etc., and now it has a new feature where I will be able to generate this list and compare it to the one you fax to me. This way, the process and the data are more valid. I want to point out that this new process is not to force you to include every student selected and to not be flagged – although not getting flagged would be nice – it is to justify each exclusion in case we do get flagged. We need to be able to show that students who were excluded should have been. This type of data should also be meaningful.

27. Scheduled January 3 – 24, 2011 The Pre-Assessment Visit (PAV) 27

28. List of Materials in the NAEP 2011 Pre-Assessment Packet PAV List Instructions for the School Coordinator brochure Student Information Report SD & ELL Storage Envelopes SD & ELL Worksheets Instructions for completing SD & ELL Worksheets State-Specific SD & ELL Guidelines Summaries Teacher Listing Form National School Lunch Program memo Parent Notification letter (all versions) Sample Questions booklets (2 sets) Frameworks (1 set) Logistics Information Form NAEP Storage Envelope Writing Computer-Based Assessment Overview Brochure (WCBA schools only) 28 Your packet will arrive via FedEx with a red NAEP sticker on it.Your packet will arrive via FedEx with a red NAEP sticker on it.

29. When You Receive the Student List (Beginning Jan. 3rd) 29 Send home the PARENT LETTER and any necessary translations at least two weeks before assessment date.(http://www.ride.ri.gov/assessment/naep.aspx.) Ensure that all THREE TASKS are completed before the PAV SD/ELL Worksheets Enrollment List Logistics Questionnaire Give copy of the Parent Letter to the AC Email the date you sent (or will send) the Parent Letter to heather.heineke@ride.ri.gov Ensure that the AC has a space in your school for at least one hour before the PAV to finish paperwork.

30. 1. Students with Disabilities & ELL Worksheets 30 Questionnaires are included in PAV packet with student names on them. Need to have Accommodations Guidelines available Need to have special education teacher and/or ELL teacher Must have approval to exclude students Questionnaires are included in PAV packet with student names on them. Need to have Accommodations Guidelines available Need to have special education teacher and/or ELL teacher Must have approval to exclude students

31. ELL Worksheet: Column P 31 In order to provide you with more guidance about completing the worksheets, ELL specialists from the WIDA states recommend using the these proficiency levels when completing the worksheet.In order to provide you with more guidance about completing the worksheets, ELL specialists from the WIDA states recommend using the these proficiency levels when completing the worksheet.

32. 2. Updated list of current students in selected grade 32 Include all students in tested grade (4th or 8th): First name, last name (alpha by last name) Birth date AC will compare the enrollment roster from October 1st and the current roster and then select new students to participate in NAEP, if necessary. One hour before the PAV

33. 3. Complete the Logistics Questionnaire 33 Must be done before the PAV Asks important logistical questions that will help the assessment team prepare

34. After the PAV 34 NOTIFY teachers and students MAIL PARENT LETTERS (if you haven’t already) can be mailed out to students at any time prior to the PAV and up to two weeks before the assessment date. Email the date you sent it to heather.heineke@ride.ri.gov SECURE the NAEP Storage Envelope 1- Any room changes, schedules, passes needed. 2- Only parent letter in packet/downloaded from website can be used. NO opt-in or opt-out forms may be used.1- Any room changes, schedules, passes needed. 2- Only parent letter in packet/downloaded from website can be used. NO opt-in or opt-out forms may be used.

35. Parent Letter (in your PAV packet) RULES Can use only this letter Send at least two weeks before assessment Snail mail newsletters DATED copy on school letterhead given AC during PAV EMAIL heather.heineke@ride.ri.govwith the date the letter(s) were sent. TRANSLATIONS Spanish Portuguese Arabic Chinese Cambodian/Khmer Hmong http://www.ride.ri.gov/assessment/naep.aspx 35 Snail mail, newsletters, attaching to progress reports, notices, or grades are all acceptable ways to send the parent letter home.Snail mail, newsletters, attaching to progress reports, notices, or grades are all acceptable ways to send the parent letter home.

36. Computer-based Assessments, NIES Special Studies 36

37. Special Studies 37 National Indian Education Study (NIES) Trying to examine the current status of Indian Education in the US Additional survey that lasts about 10-15 minutes and is taken by Native American and Alaskan Native students. Your AC will discuss the specific logistics and any other requirements with you at the PAV.

38. Preparing for Computer-Based Assessments 38 ONLY 8 schools selected. 8th GRADE Writing and Mathematics 14-15 students NAEP will bring all necessary equipment. REQUIRED: Need a room with tables for laptops. HELPFUL: Someone to help move the computer cases. To EXCLUDE students, use the Exemption Request Form

39. Design of Computer-Based Tests These features are part of the test design and are not considered accommodations. Adjusting the font size Small group One-on-one Test items read aloud occasionally (English) Test items read aloud most or all (English) Using a computer to respond Adjusting contrast colors Highlighter Electronic Spell-check and thesaurus 39

40. Accommodations for Computer-based Tests Extended time Breaks Bilingual dictionary Directions read aloud in Spanish Aide/classroom teacher in room Responds orally to scribe Magnification It is encouraged that teachers remain with their students if at all possible. 40

41. Heather Heineke 222-8493 Heather.heineke@ride.ri.gov Questions 41