Are we testing what we think is important? An exploratory, historical analysis of the

1. Are we testing what we think is important? An exploratory, historical analysis of the New York Regents Exam in Earth science Anne E. Egger, Michael Alvernaz, Ryan Carrington, and Anthony Perez Central Washington University You live in Washington – why do you care about New York? Do we even agree on what is important? Who is “we”, anyway?

2. Background: The literacy documents 192 literacy elements

3. Background: New York Regents Exam • Longest running high school science assessments (since 1878) • Only (?) Earth science exam at the high school level • Since 2001, all high school students take five regents exams, one in science • Four options in science: • Living environment • Physical setting: Earth Science • Physical setting: Chemistry • Physical setting: Physics

4. Distribution of students who take exams

5. Methods • Coded one test per decade for • Question type (MC, MC-set, SA, TF, D, etc.) • Blooms level (1-5) • Use of the reference tables (y/n) • Quantitative skills • “Sphere” (Atmo-, Hydro-, Litho-, Solar System, Universe, N/A) • Literacy element • Used an iterative process by which we compared inter-rater reliability for four exams, developed some rules • Cross-checked sphere coding with Kastenset al., 2014 • Used June exams, generally assumed to be the most consistent • Coded every June exam from 2010 to 2016

6. Results: Overall • Reminder: 192 total literacy elements in four documents • Since 1942: • Atotal of 77 elements addressed; • 41 more than twice • 29 more than five times (15%) • In a given year: • 20-33 elements addressed • An average of 24 elements per year

7. Results: By question • Other: • Topographic maps • “Experiments” • Rock/mineral ID ~15% astronomy questions

8. Results: Earth Science Literacy

10. Within ES2: Earth is 4.6 billion years old ES2.1: Earth’s rocks and other materials provide a record of its history. Earth scientists use the structure, sequence, and properties of rocks, sediments, and fossils to reconstruct events in Earth’s history. Decay rates of radioactive elements are the primary means of obtaining numerical ages of rocks and organic remains. Understanding geologic processes active in the modern world is crucial to interpreting Earth’s past.

11. Results: By question

12. Results: Atmospheric Science

14. Within AS3: Atmospheric circulations transport matter and energy AS 3.3: Patterns of circulation in Earth's atmosphere can be observed at many different spatial scales from global to local. Temperature differences, the rotation of Earth on its axis, and the configuration of the continents and oceans establish the large-scale atmospheric circulation.

15. Results: Climate and ocean literacy CS1.c: The tilt of Earth’s axis relative to its orbit around the Sun results in predictable changes in the duration of daylight and the amount of sunlight received at any latitude throughout a year. These changes cause the annual cycle of seasons and associated temperature changes. OS1.c: Throughout the ocean there is one interconnected circulation system powered by wind, tides, the force of the Earth’s rotation (Coriolis effect), the Sun, and water density differences. The shape of ocean basins and adjacent land masses influence the path of circulation. This ‘global ocean conveyor belt’ moves water throughout all of the ocean’s basins, transporting energy (heat), matter, and organisms around the ocean. Changes in ocean circulation have a large impact on the climate and cause changes in ecosystems.

16. Noticeably lacking in... Systems and systems thinking

17. Noticeably lacking in... Earth and human activity

18. What does this mean? • New York adopted the NGSS in December, 2016, and will be transitioning over the next few years • Regents exams will still be the high school assessments, presumably with modification • We can support professional development and assessment item development to build skills in systems and interactions between Earth and human activity