Philip M. Sadler, Director Science Education Department

1. Preparing for Success in College Science: The Dance of Mathematics, Misconceptions, Teacher Knowledge, and the Advanced Placement Program Philip M. Sadler, Director Science Education Department Harvard-Smithsonian Center for Astrophysics, Cambridge, MA

2. Why someone from theHarvard-SmithsonianCenter for Astrophysics?

3. Harvard-Smithsonian Center for Astrophysics • Largest astronomical research institution in the world • A partnership between: • Harvard’s Department of Astronomy • Harvard College Observatory • Smithsonian Astrophysical Observatory • More than 250 scientists in a staff of over 800 • Telescopes on earth and in space • Search for earth-like planets MSP Regional Conference

4. Education MIT B.S. in Physics Harvard Ed.M., Ed.D.’92 Teaching middle school science and math Harvard University Astronomy Ed Research Ed Methods 200 teachers Developer of Starlab Planetarium Project STAR MicroObservatory Publications 5 texts 38 papers and book chapters 4 award-winning videos Editorial Board 2 ed journals Sponsored research $56m, $5m/yr 40 staff Honors ASP Brennan Prize Project ASTRO Education Award 3 AIP Computers in Physics 1999 JRST Award Why listen? MSP Regional Conference

5. How do you rigorously measure the conceptual understanding of teachers and students in science?

6. How do you rigorously measure the conceptual understanding of teachers and students in science?

7. Psychological Foundations “The unlearning of preconceptions might very well prove to be the most determinative single factor in the acquisition and retention of subject-matter knowledge.” David Ausubel 1978 MSP Regional Conference

8. Psychological Foundations “The unlearning of preconceptions might very well prove to be the most determinative single factor in the acquisition and retention of subject-matter knowledge.” David Ausubel 1978 MSP Regional Conference

9. Clinical Interviews Minds of Our Own consists of 3-one hour programs broadcast on PBS in 1997-98. It explores the ideas of students as they come to understand scientific concepts On-on-one with students A Private Universe documents students’ ideas through their own drawings and explanations www.learner.org MSP Regional Conference

10. Students and teachers have preconceptions • Exist prior to instruction • At odds with accepted scientific thought, “misconceptions” • Commonly held, not idiosyncratic • embedded in larger knowledge structures, not just an “error” • resistant to change MSP Regional Conference

11. MOSART – Misconception Oriented Standards-based Assessment Resource for Teachers Our criteria for conceptual understanding – Students and teachers must: • Prefer accepted scientific explanations over widely-held misconceptions • Apply their knowledge to make accurate predictions MSP Regional Conference

12. Our criteria for conceptual understanding Students and teachers must: • Prefer accepted scientific explanations over widely-held misconceptions • Apply their knowledge to make accurate predictions For assessments to do this, test items must: • Include the scientifically correct answer • Include the most popular misconceptions • Be easy to score and use • Value predictive over “why” questions MSP Regional Conference

13. 5-8 Physical Science: Motions and Forces • The motion of an object can be described by its position, direction of motion, and speed. That motion can be measured and represented on a graph. MSP Regional Conference

14. The Problem 108. Kevin starts walking from a store a certain distance from his home. Which sentence is a correct description of Kevin’s motion as shown on the graph? MSP Regional Conference

15. The Correct Answer 108. Kevin starts walking from a store a certain distance from his home. Which sentence is a correct description of Kevin’s motion as shown on the graph? • b. He walks toward home, stops for a while, then walks away from home. MSP Regional Conference

16. The Fraction Who Choose Correctly 108. Kevin starts walking from a store a certain distance from his home. Which sentence is a correct description of Kevin’s motion as shown on the graph? • b. He walks toward home, stops for a while, then walks away from home. 30% MSP Regional Conference

17. Other answers that students give? 108. Kevin starts walking from a store a certain distance from his home. Which sentence is a correct description of Kevin’s motion as shown on the graph? • a. He walks toward home down a hill, then walks along a level path, then walks up a hill. • b. He walks toward home, stops for a while, then walks away from home. 30% • c. He walks away from home, stops for a while, then walks toward home. • d. He walks toward home down a hill, stops for a while, then walks up a hill. • e. He walks down a hill and gets trapped in a valley. MSP Regional Conference

18. Which answers do your students give? 108. Kevin starts walking from a store a certain distance from his home. Which sentence is a correct description of Kevin’s motion as shown on the graph? • a. He walks toward home down a hill, then walks along a level path, then walks up a hill. 28% • b. He walks toward home, stops for a while, then walks away from home. 30% • c. He walks away from home, stops for a while, then walks toward home. 18% • d. He walks toward home down a hill, stops for a while, then walks up a hill. 20% • e. He walks down a hill and gets trapped in a valley. 5% MSP Regional Conference

19. Research Questions • To what degree have students who completed science courses mastered the NRC standards? • At grade level • At prior grade levels • Are there patterns of strength and weakness? MSP Regional Conference

20. Research Questions • To what degree have students who completed science courses mastered the NRC standards? • At grade level • At prior grade levels • Are there patterns of strength and weakness? • Have primary, middle, and high school science teachers mastered the standards that they teach? • How well can teachers predict the knowledge state of their students (including misconceptions)? • What is the impact of professional development activities on teacher content knowledge? MSP Regional Conference

21. Mining the Research Literature 76. An electric cord runs from a wall outlet along the floor to a lamp. The lamp’s light is on. You carefully stack books, one at a time, on top of each other on the wire until you have 100 pounds of books. Assuming the wire does not break, what do you think would happen to the brightness of the light? The brightness of the light would decrease gradually as more books were added to the stack. The light would dim all at once at some point, then remain dim. The light would go out as soon as the first book was placed on the wire. The light would flicker or briefly dim as each book was added, then return to normal. The light would not change in brightness. 14% MSP Regional Conference

22. Mining the Research Literature 76. An electric cord runs from a wall outlet along the floor to a lamp. The lamp’s light is on. You carefully stack books, one at a time, on top of each other on the wire until you have 100 pounds of books. Assuming the wire does not break, what do you think would happen to the brightness of the light? The brightness of the light would decrease gradually as more books were added to the stack. The light would dim all at once at some point, then remain dim. The light would go out as soon as the first book was placed on the wire. The light would flicker or briefly dim as each book was added, then return to normal. The light would not change in brightness. 14% MSP Regional Conference

23. Student Preference 76. An electric cord runs from a wall outlet along the floor to a lamp. The lamp’s light is on. You carefully stack books, one at a time, on top of each other on the wire until you have 100 pounds of books. Assuming the wire does not break, what do you think would happen to the brightness of the light? The brightness of the light would decrease gradually as more books were added to the stack. 39% The light would dim all at once at some point, then remain dim. 10% The light would go out as soon as the first book was placed on the wire. 6% The light would flicker or briefly dim as each book was added, then return to normal. 11% The light would not change in brightness. 14% MSP Regional Conference

24. Breakdown of the NRCs What concepts are the standards really asking kids to know? What are the relevant misconceptions reported in the literature? Item Construction Items (M/C for ease) that represents the standard and captures kids’ knowledge based on research protocols. Validation Are the questions accurate in terms of the science? Readable? Pilot Testing (N=100/item) selection of core items that represent the most variance Large scale sample (Physical Science Example, N=1000/item) Item characteristics for 100-200 items/domain Characterization of domains: 7,000+ students, 50+ teachers Finalization of Instruments Made available for evaluation of programs like yours Test Item Development MSP Regional Conference

25. National Data: MSP Regional Conference

26. Grade 7/8 Physical Science StudentsAfter Taking a Year of Physical Science MSP Regional Conference

27. Adding HS Chemistry and Physics Students MSP Regional Conference

28. Adding HS Science Teachers MSP Regional Conference

29. Adding MS Physical Science Teachers MSP Regional Conference

30. Teacher Content and Predictive Knowledge MSP Regional Conference

31. Patterns in Classroom Data MSP Regional Conference

32. Comparison of Item Formats MSP Regional Conference

33. Teacher Content and Predictive Knowledge Across 38 Classrooms MSP Regional Conference

34. Teacher Content Knowledge and Teacher Prediction Accuracy across 38 Classrooms MSP Regional Conference

35. Patterns For each standard at each level • Students have not achieved mastery • Teachers generally overestimate student knowledge. • Teachers know far more than their students • Teacher knowledge is a not a guarantee of student knowledge • Subject do much better on items if misconceptions are not a choice • Teachers’ knowledge of student ideas is associated with higher performance than content knowledge MSP Regional Conference

36. Patterns in Professional Development Data MSP Regional Conference

37. Which factors predict teacher content knowledge of the curriculum concepts? • Grade level • Gender • Years Teaching • Years Teaching science subject • Certification in the science subject • Degrees (BS, BA, MS, PhD) • Grad Courses taken in domain • Professional development in science teaching/content MSP Regional Conference

38. Model B 36% of variance Source df ∑sq F-ratio Prob Const 1 57.56 6836.20 ≤ 0.0001 Grade band 2 0.14 8.70 0.0004 Gender 1 0.06 8.29 0.0052 Years Teaching subject 1 0.08 10.58 0.0017 Certification 1 0.03 4.49 0.0374 Yrs Teaching subject * Cert 1 0.06 7.90 0.0063 Error 73 0.61 Total 79 0.95 Predicting Teacher Masterylinear models with significant factors MSP Regional Conference

39. Which factors predict teacher content knowledge of the curriculum concepts? • Grade level • Gender • Years Teaching • Years Teaching science subject • Certification in the science subject • Degrees (BS, BA, MS, PhD) • Grad Courses taken in domain • Professional development in science teaching/content MSP Regional Conference

40. Interaction of Years Teaching Subject and Certification MSP Regional Conference

41. 2-Week Astronomy Institute • Basics • To boost astronomy background • General astronomy test • Speakers • Activities • Observing MSP Regional Conference

42. 2-Week Astronomy Institute • Basics • To boost astronomy background • General astronomy test • Speakers • Activities • Observing MSP Regional Conference

43. 2-Week Astronomy Institute • Moderate initial knowledge • Gains at all levels of teacher knowledge • Few teachers with no or negative growth MSP Regional Conference

44. 1-Week Astronomy Institute • Instrumentation • Earth-Sun connection only • Only relevant items • Speakers • Activities • Observing MSP Regional Conference

45. 1-Week Astronomy Institute • Learn to use professional instrumentation • Disciplinary domain focus • Speakers MSP Regional Conference

46. 1-Week Astronomy Institute • High initial knowledge • No gains at highest level of teacher knowledge • Many teachers with no or negative growth MSP Regional Conference

47. Comparison of 2 MSP Institutes MSP Regional Conference

48. Patterns • Some teacher content weakness at all grade levels: weakest at MS levels • Content knowledge grows very slowly for the non-certified teacher • Professional development can make a difference in teacher content knowledge • Length of program • Focus on content knowledge at grade level vs. “science apprenticeships” • Must evaluate the fulfillment of goals • Content knowledge at higher levels does not translate to knowledge at lower levels MSP Regional Conference

49. Seeking Research Partners • Professional Development • Increase in teacher content knowledge • Increase in teacher pedagogical content knowledge • Customized assessment instruments • Linking to Student Pre-Post Assessment • Curricular and Pedagogical Innovation • Impact of professional development • Teacher Subject Matter Knowledge • Accuracy of Teacher Prediction • Breakout session tomorrow MSP Regional Conference

50. Overview of Research $3M, 4-year IERI study to investigate the kinds of high school courses that best prepare college students for: • introductory courses in biology, chemistry, or physics MSP Regional Conference