Dec. 11, 2012, Sung- Youn Choi

1. Dec. 11, 2012, Sung-Youn Choi Post-doc of CREATE 4 STEM Institute, Michigan State University Students Learning in the IQWST & LPImproving middle school students’ understanding of core science ideas using coherent curriculum

2. What will we do today? • Examine why we should redesign curriculum materials • Introduce the IQWST curriculum design; the next generation of middle school science materials • Introduce the test design; aligned with a LP approach • Discuss key design principles and the CCD process • Share some of our findings; using IRT analysis • Discuss about improving middle school students’ understanding of core science ideas using coherent curriculum • Investigating and Questioning our World through Science and Technology (IQWST) • Learning Progression (LP) • Construct-Centered Design Process (CCD) • Item Response theory (IRT)

3. Why redesign curriculum? • Inadequate Science Materials • Cover many topics at a superficial level • Focus on technical vocabulary • Fail to consider students’ prior knowledge • Lack coherent explanations of real-world phenomena • Provide students with few opportunities to develop explanations of phenomena • Materials lack coherence and as a result don’t focus on building understanding over time • Students need to be prepare for the world in which they live now and will live as adults • Deeper understanding of the core ideas becomes instrumental for competitiveness in a global economy, as well as for the personal well being of each citizen • Students need deeper knowledge in order to use understanding to solve problems

4. Design the Next Generation of Science Learning Environments • Solution • The Investigating and Questioning our World through Science and Technology (IQWST) • Utilizing a coherent approach for building curriculum materials across the school years • Using learning goals as a focus • Supporting students in developing understandings of the core ideas of science and scientific practices • Applying what we know about student learning • Promoting literacy practices • Engaging students in complex tasks • Using a project-based approach

5. IQWSTDesign Principles

6. Key Design Principles • Curriculum Coherence A. Inter-unit coherence • Build links across units among the grades or content areas • Provide opportunities to develop, reinforce, and use their understanding duringtheir three-year IQWST experiences B. Intra-unit coherence • Use driving-questions as support to link ideas together • Create alignment by iteratively aligning learning goals with tasks C. Learning-goals coherence • Use big ideas • Unpack standards from a learning perspective • Create learning performances as a way to specify knowledge in use

7. A. Inter-unit Coherence: Structure of IQWST Units The IQWST is built on scientific core ideas.

8. A. Inter-unit Coherence: Scope and Sequence The core ideas are coordinated across the grades and the contents.

9. B. Intra-Unit Coherence: Driving Questions • Driving Question: • Links content in a context • Links tasks, investigations, and assessments to learning goals • Ties the unit together • Builds intra-unit coherence • Uses rich and open-ended questions with everyday language

10. C. Learning Goals Coherence • Developing Learning Goals • The construct-center design approach (CCD) • Step 1: Define the constructs • Correcting ideas and concepts to learn • Identifying potential students difficulties and alternative ideas • Step 2: Unpack the ideas and concepts • Creating a set of claims • Specifying evidences • Step 3: Design tasks and materials • Step 4: Review Products

11. IQWSTExample lesson

12. IQWST Smell Unit • 8-week, project-based unit for 6th grade students • Driving Question: How can I Smell Things from a Distance? • Three Learning Sets, 15 Lessons • Learning Set 1: Students construct models to help them understand the particle nature of matter while focusing on the behavior of gases • Learning Set 2: How models help to explain why different materials have different properties • Learning Set 3: Using the particle nature of matter model to explain phase changes  Ideas from all three learning sets are brought together through a culminating final task in which students use their knowledge of the particle nature of matter.

13. Driving question for Unit Scientific Core Idea Particle Nature of Matter Properties of Matter Phase Changes

14. Example (1) activity • Learning Goal: Lesson 13 • Students will explain phase changes from gas to liquid to solid and from solid to liquid gas at the molecular level. • Scientific Principle • Evaporation

15. Example (2) reading Everyday experiences Driving question Prior knowledge Question & writing opportunity Set purpose for reading • Scientific Principle • Condensation • Alternative Conceptions • Condensation on the outside of a container is water that seeped through the container itself (or sweated through the walls of the container). • The coldness comes through the container and produces water. • Condensation is when air turns into a liquid.

16. Example test items

17. Example item (#1) PC02 • Contents area • Phase change, Structure • Claim • Students can explain how condensation occurs (without a particulate model). • Unpacking/evidence • Level 1. Matter can exist as a solid, liquid or gas. Generally, the same pure substance can exist in all three states. The state of matter depends on the temperature.

18. Example item (#1) PC02 • Contents area • Phase change, Structure • Claim • Students can explain how condensation occurs (without a particulate model). • Unpacking/evidence • Level 1. Matter can exist as a solid, liquid or gas. Generally, the same pure substance can exist in all three states. The state of matter depends on the temperature.

19. Example item (#2) PC18 • Contents area • Phase change, Conservation • Claim • Students can explain the mass of matter are conserved in transforming from solid to liquid or solid to gas (without a particulate model). • Unpacking/evidence • Level 1. Matter is not created or destroyed when substances change form or change into other substances (e.g., crushing, dissolving, phase change, chemical reaction).

20. Example item (#2) PC18 • Contents area • Phase change, Conservation • Claim • Students can explain the mass of matter are conserved in transforming from solid to liquid or solid to gas (without a particulate model). • Unpacking/evidence • Level 1. Matter is not created or destroyed when substances change form or change into other substances (e.g., crushing, dissolving, phase change, chemical reaction).

21. Example item (#3) PC19 • Contents area • Phase change, Conservation • Claim • Students can explain how condensation occurs (without a particulate model). • Unpacking/evidence • Level 1. Matter is not created or destroyed when substances change form or change into other substances (e.g., crushing, dissolving, phase change, chemical reaction).

22. Example item PC02, PC18, PC19 • Can we give same credit to each item? • Do students need to take every item?

23. Item Response Theory: IRT analysis vs. CTT (classical test theory)Creating test forms

24. A. Scoring system • CTT (raw score) • O X O OO = 4 (total score) • How many correct answers did you get? • The original result obtained by a student on a test. (% of correct answer, mean score, total score, or etc.) • IRT (latent ability score/logit) • ? ? ? ? ? = 1.3 • What is the probability of a correct response to an item? • Using the IRT model (latent trait models), we can estimate overall latent ability score/ logit. • In three parameter logistic model (3PL) • difficulty • discrimination • pseudo-guessing

25. Example: PC02 (ice cup, condense) Observed response Item difficulty = 0.958 Latent trait model The item difficulty is determined by the latent trait logit of a person who has 50% of right answer. In this graph, 0.5 probability meets the trait graph at 0.958. The MNSQ fit is around 1. It has good fitness between observed responses and latent trait model.

26. Example: PC18 (solid  gas) Observed response Item difficulty = - 0.253 Latent trait model The item difficulty is determined by the latent trait logit of a person who has 50% of right answer. In this graph, 0.5 probability meets the trait graph at -0.253. MNSQ fit is around 1. It has good fitness between observed responses and latent trait model.

27. B. Test form • CTT • (missing data) O X O OO _ _ = 4, or deleted • The raw score, which is the number of items correct, does not present a broad picture of test performance because it can be interpreted only in terms of a particular set of test questions. • IRT • (missing data) ? ? ? ? ? _ _ = 1.7 • Linking item • Latent student ability score allows direct comparisons of student performance between specific sets of test items. • IRT generally brings greater flexibility and provides more sophisticated information.

28. B. Test form (LP test design) • Learning Progressions • LPs are research-based descriptions of how students build their knowledge within and across disciplines (topics) over a broad span of time • Explaining many important phenomena requires incorporating ideas from many topic areas • Focusing on the growth of sets of ideas instead of individual topic Phase Change Chemical Reaction Others Conservation Structure of matter Topic areas Interactions Phenomena

29. B. Test form (LP test design) • Learning Progressions • LPs are research-based descriptions of how students build their knowledge within and across disciplines (topics) over a broad span of time • Explaining many important phenomena requires incorporating ideas from many topic areas • Focusing on the growth of sets of ideas instead of individual topic Phase Change Chemical Reaction Others Conservation Structure of matter Topic areas Interactions Level 2 Level 3 Level 4 Level 5 Level 6 Level 1 Phenomena

30. B. Test form (LP test design) • Test form design • Total 62 items; 52 multiple-choice items and 10 open-ended or short-answer items • A-version Form (1A, 2A, 3A 4A) • Each containing 14 items • For 6th, 7th grade • B-version From (1B, 2B, 3B, 4B) • Each containing 15 items • For 8th grade • Three items were used to adjust the difficulty between A- and B-version • Linking items • Each test form contained three pairs of linking items • Total six linking items were contained • Linking items cover a range of contents areas along levels 1 to 3 on the LP

31. Research Questions &Findings

32. Research Questions • Do the chemistry IQWST units, which link content within and across units, build understanding of targeted chemistry ideas? • Comparing students’ performances and learning gains in Phase Change and Chemical Reaction • Comparing Students performances among Topic Areas • Do student apply what they learned from IQWST Units to solve new problems? • Comparing between IQWST items and non-IQWST items in Chemical Reaction

33. Data Collection: pre- and post-test • Participants • Total 1,519 students • 481 students in 6th grade, 519 in 7th grade, 519 in 8th grade • 7 teachers • 6 schools • Two of the schools represent middle- to upper-middle class socioeconomic status • Four of the schools were from neighborhoods that are lower-SES, with a majority of students eligible for free and reduced lunch. • Research period • Schools used IQWST materials during national field trials (2007~2010) • Pretest (fall, 2010), Posttest (Spring, 2011)

34. Test Instrument • Hypothetical LP based test • We developed test instruments based on a Learning Progression (LP) for the transformation of matter • The assessment instruments measured how students applied ideas within and across topics to explain phenomena involving transformations of matter. • Test forms • total 62 items, 8 different forms, each form contained 3 pairs of linking items for a total of 6 linking items. • Validation of instrument • Piloted with students from grade 6th-15th (N=792). • Content validity: semi-structured interviews with a subset of students supplemented the pilot test data. • The LP research team is contributing to validate instruments. (Namsoo Shin, Shawn Stevens)

35. Framework (1) • Phenomenon • Phase Change: 6th grade IQWST unit, 21 test items (20 with IQWST) • Chemical Reaction: 7th grade IQWST unit, 16 test items (11 with IQWST) Phase Change Chemical Reaction Others Conservation Structure of matter Topic areas Interactions Phenomena

36. Framework (2) • Topic areas • Conservation • Structure of matter; matter and materials, properties and periodicity, atomic structure • Interactions; kinetic energy, forces, energy, processes Phase Change Chemical Reaction Others Conservation Structure of matter Topic areas Interactions Phenomena

37. Item analysis • The purpose of item analysis is to provide how well assessments work, and how well individual items on assessments work. • Item Response Theory (IRT, Wu, et al., 2007) • IRT allows us to evaluate students’ ability and to describe how well items on the test are performing. In IRT, ability and item parameters are both estimated based on students’’ response patterns on the test. • Item parameters (difficulty, discrimination, pseudo-guessing)were used to determine whether an item displayed sound psychometric properties • General item analysis criteria • Items difficulty (estimate) • MNSQ fit (the weighted means square) ~ 1 • T (pseudo-guessing) <3 • Item correlation > 0.3 • % of correct answer (total)

38. Analyzing data set • Phase Change (20 IQWST items + 1 non-IQWST item) • Conservation (4) • Structure of matter (8+ 5 overlapped with interactions + 1 non-IQWST) • Interactions (3+ 5 overlapped with structure) • Chemical Reaction (11 IQWST items + 5 non-IQWST items) • Conservation (6+ 2 overlapped+ 1 non-IQWST) • Structure of matter (2+ 2 overlapped+ 5 non-IQWST) • Interactions (2+ 1 non-IQWST)

39. Result fromPhase Change

40. Item analysis (PC) Problematic item • Accepted items range • - 0.988 < d <1.174 • MNSQ fit<1.17 • - 4.0 < T < 3.0 • 0.35 < r

41. Item difficulties & Latent distribution (PC) conservation structure Interactions Students distribution Item diff distribution conservation structure Interactions

42. Scoring • Students latent ability logit • The partial credit, multi dimensional model • The expected a posteriori (EAP) parameter was standardized to present student ability score • Students ability score • To facilitate the interpretation of their ability scores assigned to students • Normalization as t2 so that the mean and S.D of scores was 500 and 100

43. R1. Do students develop understanding of Phase Change? • Phase Change; Topic- 1. Conservation 6th grade Pretest Posttest 7th grade Pretest Posttest 8th grade Pretest Posttest Students’ ability score Distribution of Students

44. R1. Do students develop understanding of Phase Change? • Phase Change; Topic- 1. Conservation % of variance = 0.46 ** % of variance = 0.45 ** % of variance = 0.22 IQWST curriculum

45. R1. Do students develop understanding of Phase Change? • Phase Change; Topic- 2. Structure of Matter 6th grade Pretest Posttest 7th grade Pretest Posttest 8th grade Pretest Posttest Students’ ability score Distribution of Students

46. R1. Do students develop understanding of Phase Change? • Phase Change; Topic- 2. Structure of Matter % of variance = 0.63 ** % of variance = 0.45 ** % of variance = 0.28 **

47. R1. Do students develop understanding of Phase Change? • Phase Change; Topic- 3. Interactions 6th grade Pretest Posttest 7th grade Pretest Posttest 8th grade Pretest Posttest Students’ ability score Distribution of Students

48. R1. Do students develop understanding of Phase Change? • Phase Change; Topic- 3. Interactions % of variance = 0.48 ** % of variance = 0.49 ** % of variance = 0.37 **

49. Result fromChemical Reaction

50. Item difficulties & Latent distribution: (2) CR conservation structure Interactions conservation structure Interactions Students distribution Item diff distribution