Assessment and Instruction Aligned to Science Content Standards George E. DeBoer Deputy Director, AAAS Project 2061

1. Assessment and Instruction Aligned to Science Content StandardsGeorge E. DeBoerDeputy Director, AAAS Project 2061 Association of American Publishers Summit on Math and Science Education October 4, 2007 Arlington, VA This work is funded by the National Science Foundation ESI 0352473

2. Project 2061: Student Assessment 2

3. Project 2061: Student Assessment 3

4. Project 2061: Student Assessment 4

5. Project 2061: Student Assessment 5

6. Project 2061: Student Assessment 6 We have also developed nearly 100 K-12 strand maps that show conceptual connections among benchmarks on a topic both within grade spans and across grades. At the state and school district level, educators have long recognized the importance of K-12 articulation in the science curriculum, but it is still unusual for school districts to have a coherent and articulated curriculum. The Atlas maps point the way to a curriculum plan that can accomplish this goal. Atlas maps are now being used by states and districts to guide the development of an articulated curriculum, to revise content standards and curriculum frameworks, and to inform the development of assessments.We have also developed nearly 100 K-12 strand maps that show conceptual connections among benchmarks on a topic both within grade spans and across grades. At the state and school district level, educators have long recognized the importance of K-12 articulation in the science curriculum, but it is still unusual for school districts to have a coherent and articulated curriculum. The Atlas maps point the way to a curriculum plan that can accomplish this goal. Atlas maps are now being used by states and districts to guide the development of an articulated curriculum, to revise content standards and curriculum frameworks, and to inform the development of assessments.

7. Each map in Atlas shows a K-12 progression on a topic. The boxes contain the text of benchmark ideas for K-2, 3-5, 6-8, and 9-12; and the arrows show which ideas contribute to which others. The maps are organized around a small number of conceptual strands, shown at the bottom. The strand on the WEATHER AND CLIMATE map are temperature and winds, water cycle, atmosphere, and climate change. The facing page provides a brief introduction to the map that describes its importance to science literacy, characterizes the nature of the learning across K-12, and summarizes the research on student learning that contributed to the development of the map.Each map in Atlas shows a K-12 progression on a topic. The boxes contain the text of benchmark ideas for K-2, 3-5, 6-8, and 9-12; and the arrows show which ideas contribute to which others. The maps are organized around a small number of conceptual strands, shown at the bottom. The strand on the WEATHER AND CLIMATE map are temperature and winds, water cycle, atmosphere, and climate change. The facing page provides a brief introduction to the map that describes its importance to science literacy, characterizes the nature of the learning across K-12, and summarizes the research on student learning that contributed to the development of the map.

8. Project 2061: Student Assessment 8 If we zoom in on the high school region of the map we see a benchmark describing the contribution of greenhouse gases to climate change, an idea feeding into it that relates greenhouse gases to increasing earth�s temperature and another benchmark that describes the role of plants in removing carbon dioxide from the atmosphere. Note that our familiar benchmark about plant food making is shown as an off map connection that contributes to students� understanding of the role of plants in changing the earth�s atmosphere. If we zoom in on the high school region of the map we see a benchmark describing the contribution of greenhouse gases to climate change, an idea feeding into it that relates greenhouse gases to increasing earth�s temperature and another benchmark that describes the role of plants in removing carbon dioxide from the atmosphere. Note that our familiar benchmark about plant food making is shown as an off map connection that contributes to students� understanding of the role of plants in changing the earth�s atmosphere.

9. Project 2061: Student Assessment 9 Two main thrusts of our current work The development of tools and resources to support effective goals-based instruction The development of tools and resources to support effective goals-based assessment

10. Project 2061: Student Assessment 10 We begin by writing clarification statements to specify what students should know and be able to do. (Standards by themselves tend to be under-specified.) In 2001, the Commission on Instructionally Supportive Assessment said: �A state�s high priority content standards must be clearly and thoroughly described so that the knowledge and skills students need to demonstrate competence are evident. [This] should result in relatively brief, educator-friendly descriptions...�

11. Project 2061: Student Assessment 11 Benchmarks, key ideas, and clarifications Atoms and molecules are perpetually in motion. Increased temperature means greater average energy of motion, so most substances expand when heated. (Benchmark 4D/M3ab)

12. Project 2061: Student Assessment 12 Key Idea 1: Atoms and molecules are perpetually in motion. Students should know that atoms and molecules of all matter are always moving.�They are expected to know that this is true for atoms or molecules of solids, liquids, and gases.� They are expected to know that, even when objects that are made up of these atoms and molecules appear not to be moving, the atoms and molecules that make up those objects are nonetheless themselves in constant motion. Students should know that the motion of atoms or molecules can include moving back and forth with respect to a fixed point, around a fixed point, and/or past each other from one fixed point to another�

13. Project 2061: Student Assessment 13 Atoms/Molecules of Solids and Liquids in Motion There is a solid wooden table with a cup of water sitting on it. Which of the following statements about the atoms and molecules of the table and the atoms and molecules of the water is TRUE? A. The atoms and molecules of both the liquid water and the table are moving.* B. The atoms and molecules of both the liquid water and the table are not moving. C. The atoms and molecules of the liquid water are not moving, and the atoms and molecules of the table are moving. D. The atoms and molecules of the liquid water are moving, and the atoms and molecules of the table are not moving.

14. Project 2061: Student Assessment 14 Atoms/Molecules of Solids and Gases in Motion A balloon full of air is placed on a chair. Which of the following statements about the atoms and molecules of the chair and the atoms and molecules of the air in the balloon is TRUE? A. The atoms and molecules of both the chair and the air in the balloon are moving.* B. The atoms and molecules of both the chair and the air in the balloon are not moving. C. The atoms and molecules of the chair are not moving, and the atoms and molecules of the air in the balloon are moving. D. The atoms and molecules of the chair are moving, and the atoms and molecules of the air in the balloon are not moving.

15. Project 2061: Student Assessment 15 Results: Molecules of gases in motion: 79.3% Molecules of liquids in motion: 69.7% Molecules of solids in motion: 43.5%

16. Project 2061: Student Assessment 16 Key Idea: Increased temperature means greater average energy of motion so most substances expand when heated. Students should know that as the temperature of a substance increases the distance between the atoms and/or molecules of the substance typically increases and the substance expands. Students should know the reverse of this expansion occurs when the temperature of a substance is decreased. They should know that as the temperature decreases, the distance between the atoms and/or molecules decreases and the substance contracts. Students are expected to know that this expansion or contraction can happen to solids, liquids, and gases�

17. Project 2061: Student Assessment 17 Sample Assessment ItemIron Frying Pan Item (Atoms-only version) After cooking breakfast, a cook places a hot iron frying pan on the counter to cool.� What happens as the iron pan cools? The iron atoms get heavier. The iron atoms decrease in size. The number of iron atoms increases. The distance between iron atoms decreases.* Here is a sample assessment item that probes student understanding of what happens to an iron frying pan when it is heated. Distractors were chosen to correspond to misconceptions documented in the literature.Here is a sample assessment item that probes student understanding of what happens to an iron frying pan when it is heated. Distractors were chosen to correspond to misconceptions documented in the literature.

18. Project 2061: Student Assessment 18 Iron Frying Pan Item (Atoms + Macro-phenomenon) After cooking breakfast, a cook places a hot iron frying pan on the counter to cool.� What happens as the iron pan cools? Even though you cannot see it, the pan gets a tiny bit smaller because the iron atoms decrease in size. Even though you cannot see it, the pan gets a tiny bit smaller because the distance between iron atoms decreases.* Even though you cannot feel it, the pan gets a tiny bit heavier because the iron atoms increase in mass. Even though you cannot feel it, the pan gets a tiny bit heavier because the number of iron atoms increases. Here is a related item that probes whether students can link what is happening to the atoms to what is happening to the substance. In other words, this item is attempting to find out if students can apply what they have learned about atoms to explain phenomena in the natural world.Here is a related item that probes whether students can link what is happening to the atoms to what is happening to the substance. In other words, this item is attempting to find out if students can apply what they have learned about atoms to explain phenomena in the natural world.

19. Project 2061: Student Assessment 19 Results of Pilot Testing This slide shows results of pilot testing items on a small number of students (field tests will use a large national sample of students). As you can see about half of the students responded correctly to the atomic version�that is, half of the students knew what they learned in class. However, only 13% of the students could apply what they learned to relate what happens to the atoms (they get farther apart) to what happens to the frying pan (it expands).This slide shows results of pilot testing items on a small number of students (field tests will use a large national sample of students). As you can see about half of the students responded correctly to the atomic version�that is, half of the students knew what they learned in class. However, only 13% of the students could apply what they learned to relate what happens to the atoms (they get farther apart) to what happens to the frying pan (it expands).

20. Project 2061: Student Assessment 20 Sample Phenomenon to Illustrate Key Idea: Thermal Expansion of a Solid Students observe that a metal ball that fits through a metal ring will no longer fit through the ring after the ball is heated. Students need to interpret the �lack of fitting� as an indication of the thermal expansion of the metal ball. To help students reconcile this phenomenon with their everyday observations that macroscopic substances don�t appear to expand or contract, students need to appreciate that the ball-and-ring device is capable of detecting small changes that their eyes may not detect. As a result of these findings, we have identified two phenomena that could be helpful. This one involves a metal ball and a ring that fits snugly around it. Students observe that the ball initially slips through the metal ring but that after the ball is heated it no longer slips through the ring. Accompanying note remind teachers that students need to interpret the lack of fitting as an indication of the thermal expansion of the metal ball and that to help students relate these observations to the frying pan, they need to realize that their eyes cannot detect the ball getting bigger any more than their eyes cannot detect the frying pan getting bigger, that the ring provides a detection device that is more sensitive than their eyes.As a result of these findings, we have identified two phenomena that could be helpful. This one involves a metal ball and a ring that fits snugly around it. Students observe that the ball initially slips through the metal ring but that after the ball is heated it no longer slips through the ring. Accompanying note remind teachers that students need to interpret the lack of fitting as an indication of the thermal expansion of the metal ball and that to help students relate these observations to the frying pan, they need to realize that their eyes cannot detect the ball getting bigger any more than their eyes cannot detect the frying pan getting bigger, that the ring provides a detection device that is more sensitive than their eyes.

21. Project 2061: Student Assessment 21 Sample Phenomenon to Illustrate Key Idea: Thermal Expansion of a Liquid Students observe that the level of liquid mercury rises as a thermometer is heated. Students need to interpret the height increase of the liquid mercury as an indication of its thermal expansion. To help students reconcile this phenomenon with their everyday observations that macroscopic substances don�t appear to expand, students need to appreciate that the tiny diameter of the thermometer makes it easier to detect the change. Similarly, students will need help in recognizing that their eyes aren�t sensitive enough to detect the expansion of liquids, because liquids don�t expand very much either. However, the narrow diameter of the tube inside the thermometer makes it possible to detect the small increase in volume of the liquid. Note that these examples provide what textbooks do not�they engage students in natural phenomena and help students relate the scientific ideas to the phenomena and help students to recognize the limitations of their own ideas.Similarly, students will need help in recognizing that their eyes aren�t sensitive enough to detect the expansion of liquids, because liquids don�t expand very much either. However, the narrow diameter of the tube inside the thermometer makes it possible to detect the small increase in volume of the liquid. Note that these examples provide what textbooks do not�they engage students in natural phenomena and help students relate the scientific ideas to the phenomena and help students to recognize the limitations of their own ideas.

22. Project 2061: Student Assessment 22 Findings related to food for growth Many students think that food provides energy for growth but not materials for growth. Item: For an animal to grow, what must happen to the food that it eats? The food must be changed into energy or eliminated as waste. 42.6% The food must be broken down into simpler substances that become part of the animal�s body.* 14.8% Middle school students typically don�t think about where the mass for growth comes from. They don�t worry about conservation of mass. Current instruction focuses almost entirely on the energy story; almost nothing on the matter story. Instruction is needed to counter the idea that things just grow.

23. Project 2061: Student Assessment 23 Findings related to food web diagrams On items using food web diagrams, students have difficulty with indirect effects and with items that use symbols instead of names of organisms.

24. Project 2061: Student Assessment 24 Assessing Student Understanding of Direct Effects: ����������������������������WORMS�?��ROBINS ? FOXES����������� If a disease kills most of the worms, which of the following statements describes what will happen to the robins? 75% correct

25. Project 2061: Student Assessment 25 Assessing Student Understanding of Indirect Effects: ������������������������� WORMS�?����ROBINS ? FOXES ���������������������� If a disease kills most of the worms, which of the following statements describes what will happen to the foxes? 45% correct

26. Project 2061: Student Assessment 26 Assessing Student Understanding with Symbols for Organisms: ���������������� A�?���B ? C ������������������������������� If a disease kills most of the �A�s, which of the following statements describes what will happen to population B? 36% correct

27. Project 2061: Student Assessment 27 The Current Work on Assessment: We are creating a bank of middle school science assessment items that are precisely aligned with national content standards and that can be used for diagnostic purposes. A resource for educators

28. Project 2061: Student Assessment 28 Item Development Two-year item development cycle Clarify learning goals. Review the research literature on student learning. Design items that are content-aligned and that use misconceptions as distractors. Pilot test each item (about 150 students per item). Students are asked to give us feedback about the item. Teams of external reviewers formally evaluate items. Field test each item (about 1000 students per item�national sample).

29. Project 2061: Student Assessment 29 Assuring Content Alignment: Criteria of Necessity and Sufficiency Necessity: Knowledge of the idea described in the learning goal must be needed to evaluate each of the answer choices. Sufficiency: Knowledge of the idea described in the learning goal should be enough by itself to successfully evaluate each of the answer choices.

30. Project 2061: Student Assessment 30 Assuring Construct Validity: Items should be comprehensible and accessible to all students (use plain language, i.e., simple vocabulary and sentence structure) Task contexts should not advantage or disadvantage one group of students because of their interest or familiarity with the context. Items should not be easy to answer by using test-wiseness strategies.

31. Project 2061: Student Assessment 31 The Current Work in Instruction We are developing a data base of phenomena and representations linked to content standards, and closely linked to our work in assessment. A resource for educators

32. Project 2061: Student Assessment 32 Some Thoughts about How to Improve Science Education

33. Project 2061: Student Assessment 33 What we can do to improve science education: Focus on the most important and teachable ideas in science. We can�t teach everything. Standards point the way. Use assessment for diagnostic purposes. Assessment is not just about accountability. Connect science ideas to real-world phenomena. Phenomena alone and abstract principles alone are not effective in achieving the goal of science literacy. Embed pedagogical support into curriculum materials (e.g., to address the key misconceptions students have, to provide opportunities for students to make sense of science ideas, and to make the connections among ideas clear).

34. Project 2061: Student Assessment 34 One More Thought Can researchers and implementers work more closely together? Project 2061, for example, is a research group without an implementation arm. I assume there are publishers without a research arm. Is more collaboration possible?