Academic Rigor: Where Are We Now

1. Academic Rigor: Where Are We Now Looking at Student Reading Achievement and Increasing Rigor Using Grade 4 NAEP Item Maps and Percentile Graphs Prepared by Jeanne Foy Alaska State NAEP Coordinator

2. NAEP: the Common Measurement of Student Achievement among States • The National Assessment of Educational Progress is called the “gold standard of assessment” • Created in 1969; the No Child Left Behind Act (NCLB) mandated state participation in NAEP reading and math every other year • Alaska has NAEP data for 2003, 2005, 2007, 2009, and 2011

3. Purpose of This Presentation • NAEP reading item maps and percentile graphs can help to frame the question of where Alaska is now in terms of student achievement and what would need to be done to raise student achievement. • Alaska is revising reading, writing, and math standards against the background of a national push to raise the rigor of standards.

4. Alaska’s Revised Standards • Many states have adopted the Common Core standards for English language arts and mathematics. Alaska has developed proposed standards to equal the Common Core standards in rigor: they will be posted at http://www.eed.state.ak.us/regs/ • The next slide shows the rigor intended to be reached by the new standards

5. CC Standards Designed to Be Internationally Competitive • The standards have been informed by the best available evidence and the highest state standards across the country and globe . . . • These standards are designed to ensure that students graduating from high school are prepared to go to college or enter the workforce and that parents, teachers, and students have a clear understanding of what is expected of them. • The standards are benchmarked to international standards to guarantee that our students are competitive in the emerging global marketplace. http://www.corestandards.org/frequently-asked-questions

6. NAEP Resources Set Stage for Discussions about Higher Rigor • NAEP percentile box graphs show current academic achievement gaps by ethnic groups • NAEP item maps show what students are achieving now

7. A Disclaimer! • The issues of increasing academic performance and closing gaps are complicated • This presentation is designed to show current student achievement in Alaska and serve as a springboard for discussion in considering the challenges of increasing academic rigor and closing achievement gaps among ethnic groups

8. How Percentiles Represent Student Achievmeent • A percentile is a value on a scale of one hundred that indicates the percent of a distribution that is below it. • For example, if a student scores at the 84th percentile on a test, he scored higher than 84% of the students who took the test

9. Understanding Box Plots • Box plots allow data to be explored by showing the median value of a set of data and the 25th and 75th percentiles • The middle line in the box shows the median value, the reading score at which half the students performed below and half the students performed above • Students who scored at the 75th percentile performed better than 75% of students who took the NAEP; students who scored at the 25th percentile performed better than 25% of students who took the NAEP • The range of the distribution is shown on the box plot by showing a minimum and maximum value (the 10th and 90th percentiles for the NAEP box plots)

10. Alaska Gr. 4 Reading Percentile Box Plots for 2011 NAEP • Note where the median (the dot in the middle of the box) is for each group in relation to the other groups • Note where the other data points are in relation to the other groups • What are your observations? • What questions or issues do your observations raise?

11. NAEP Item Maps Illustrate What a Reading Score Represents • Item maps help to illustrate the knowledge and skills demonstrated by students performing at different scale scores; item maps show questions students answered correctly at different scale scores • Item maps combined with percentile box charts help to illustrate the knowledge and skills demonstrated by students performing at different scale scores

12. For instance, the item or question description at 211 on the scale represents the types of items grade 4 students who scored a 211 on NAEP answered correctly; item descriptions in blue represent items that are available to the public

13. Looking at the actual question for score 211 helps show what students can do in a concrete way. The score of 211 is close to the 75th percentile for American Indian/Alaska Native students. American Indian/Alaska Native students scoring at the 75th percentile would probably have answered this question correctly. This question is shown on the next slide.

14. 211 Recognize specific information in expository event—Locate/recall • Question*: What is the main purpose of the article? A. To describe the variety of bees and what bees do B. To explain the ways pollen is used by bees C. To show the way bees communicate with each other D. To show what different species of bees look like Nationally, 76% of students chose A (correct answer). In Alaska, 66% chose A, 17% chose B, 11% chose D. *All questions in this presentation are over the nonfiction reading passage, What’s the Buzz. The proposed English/Language Arts standards emphasize the importance of reading nonfiction.

15. Each NAEP Question Has a Cognitive Target • There are three targets--locate/recall, integrate/interpret, and critique/evaluate. • Locate/recall description: • Identify textually explicit information and make simple inferences within and across texts, such as: • Definitions • Facts • Supporting details

16. What Skills Are Required to Answer the Question • What is text complexity of the passage for fourth grade students? • What skills do students demonstrate when they answer the question correctly? • Why would students choose the wrong answer? • Consider the percentage who chose the wrong answers. • What might lead students to choose those answers?

17. Consider Another Locate/Recall Question White Black Hispanic Asian/Pacific Islander Am Indian/AK Native The question at 239 is between the 50th and 75th percentile for White students. The question is on the next slide.

18. 239 Recognize specific detail of supporting information—locate/recall • Question According to the article, what can animals of the same species do? A Travel in groups over long distances B Live together in homes such as hives C Mate with each other and give birth D Find food for their young Nationally, 63% of students chose C (correct answer). In Alaska, 57% chose C, 10% chose A, 21% chose B, and 12% chose D.

19. How Do the Two Locate/Recall Questions Differ What makes one question easier than the other? What skills are needed for students to answer the more difficult question?

20. What Does Item at Upper End of Item Map Show The question at 309 is located above the 90th percentile for any ethnic group. The question is shown on the next slide.

21. 309 Use specific information to describe and explain a process—integrate/interpret • Question • Using information from the article, explain what pollination is and how it happens. • Note: the score of 309 is for students who scored at the full comprehension level according to the scoring guide, which is presented in the following slides.

22. Integrate/Interpret • Make complex inferences within and across texts to describe problem and solution, cause and effect: • Compare or connect ideas, problems, or situations. • Determine unstated assumptions in an argument. • Describe how an author uses literary devices and text features.

23. Item: Use specific information to describe and explain a process—integrate/interpret • Consider text complexity of reading passages. • How difficult do you think this item is? • How do you think grade 4 students should perform on this item?

24. Scoring Guide for Full Comprehension Responses at this level use information from the article to explain what pollination is and how it happens. •  Pollination is the first step of making seeds. This happens when the pollen from the stamen reaches the pistil. •  Pollination is the way that plants reproduce. The way it happens is that the bees carry the pollen and it drops on the female part of the flower and it reproduces.

25. Scoring Guide for Partial Comprehension a) Responses at this level use information from the article to explain what pollination is, but they do not explain how it happens. •  It reproduces and makes new flowers. •  Pollination is the first step in making seeds and it happens right beside the pistil. OR b) Responses use information from the article to explain how pollination happens, but they do not explain what pollination is.  •  When bees get pollen and they go to another plant and that gets the pollen from the other plant so that's what pollination is. •  Pollination is the movement of pollen from one flower to another. Pollination happens when one bee rubs itself against a flower and moves to another one.

26. Scoring Guide for Little or No Comprehension Responses at this level provide incorrect information, irrelevant details, or personal opinions. Responses may simply repeat the question. •  First the bees take pollen to the hive, then turn it into honey, then the bears eat it. •  Bees get pollen on them. The word "pollen" can be taken to mean "pollination." The words "seeds" and "nectar" are not given credit for "pollen."

27. What Skills Do Students Need to Demonstrate Full Comprehension? Examples of full comprehension

28. What Is Going on with Little Comprehension? Examples of little or no comprehension

29. What Does Data Suggest about Student Skills? National performance data • In Alaska, 11% got full comprehension, 32% got partial, 50% had little or no comprehension.

30. Questions to Consider • What does data show about the current achievement gaps? • What are the challenges for increasing academic rigor for all students? • What skills do students need to increase their achievement as shown by questions on the NAEP item maps?

31. Additional Resources • The Alaska-created “NAEP Assessment Toolbox for Teachers” show how NAEP questions can be used instructionally http://www.eed.state.ak.us/tls/assessment/naep.html • NAEP questions for a variety of subjects can be found at http://nces.ed.gov/nationsreportcard/itmrlsx/default.aspx