Zdeslav Hrepic Dean A. Zollman N. Sanjay Rebello

1. Zdeslav HrepicDean A. ZollmanN. Sanjay Rebello Issues in Addressing and Representing Hybrid Mental Models AAPT, Sacramento Fort Hays State UniversityKansas State University Supported by NSF ROLE Grant # REC-0087788

2. Goal of the study To create a multiple choice test… …that can elicit students’ mental models of sound propagation… …during the lecture… …using a class response system and appropriate software.

3. Real time, in class assessment Enables quick collection and immediate analysis of students responses in the classroom. Uses some form of Class Response System

4. Benefits of class assessment • Engages students. • Facilitates interactive learning and peer instruction (especially in large enrolment classes). • Gives immediate feedback to the teacher. • Enables the teacher to adjust the teachingbeforethe exam rather than after it and according to specific needs of his/her students. • Allows a post lecture detailed analysis.

5. Research questions • Main question: • What is the optimal multiple choice test that can elicit students’ mental models of sound propagation in a real time, during the instruction? • Sub questions (Addressed in the presentation): • What is the optimal analytical tool for analysis of students’ responses in this test? • How do we represent data so the display provides a variety of instruction guiding information?

6. Starting point in test creation:Identifying mental models of sound propagation • Wave Model - Scientifically accepted model • Independent Entity Model - Dominant alternative model: Sound is a self-standing, independent entity different from the medium through which it propagates. • Hybrid models - Composed of entity and wave model features and at the same time they are incompatible with both the entity and the wave models. (E.g.) Hrepic, Z., Zollman, D., & Rebello, S. (2002). Identifying students' models of sound propagation. Paper presented at the 2002 Physics Education Research Conference, Boise ID.

7. Mental models of Earth Mixed ModelState Initial model Target model Hybrid Models

8. MetaphorMental Models and Model states Horse Hybrid = Mule Donkey A mule = hybrid of a donkey + a horse. A horse –64 chromosomesA donkey – 62 chromosomesA mule – 63 chromosomes http://www.luckythreeranch.com/muletrainer/mulefact.asp

9. Model States(In terms of children’s mental models of Earth; Vosniadou, 1994) Pure Model 1 State Pure Model 2 State Mixed ModelState Hybrid Model State Instance1 Instance2

10. Hybrid mental models identified in domains of… • Earth science(Vosniadou, 1994) • Electrostatics(Otero, 2001) • Newtonian mechanics(Hrepic, 2002; Itza-Ortiz, Rebello, & Zollman, 2004) • Sound(Hrepic, 2002; Hrepic, Zollman, & Rebello, 2002). • Optics(Galili, Bendall, & Goldberg, 1993) (“hybridized knowledge”) • Inertia and gravity(Brown & Clement, 1992) (“intermediate concepts”)

11. Implications of hybrid mental models Implications for teaching • A student can give a variety of correct answers on standard questions using a hybrid (wrong) model. Implications for analysis of our test • Hybrid models cause overlaps in multiple choice questionnaires – more than one model corresponds to the same choice. • Complexity: 3 questions define a model • Model analysis requires that each answer choice is uniquely associated with a model.

12. Model States Knowledge elements related to Model 1 only Knowledge elements related to Model 2 only x Knowledge elements related to both models or neither one NoModelState Pure Model 1 State Pure Model 2 State Mixed ModelState Hybrid Model State x x x x x Instance1 x x x x x x x x x x x Instance2 x x x x x x x

13. Human characters = Air particles Footballs = Sound entities 4 basic models - mechanisms of propagation

14. 4 basic models - mechanisms of propagation WaveModelScientifically Accepted Model (+) Ear Born Sound Propagating Air Hybrid Models Dependent Entity Independent EntityDominant AlternativeModel

15. Test Contexts1. Air How does sound propagate in this situation?

16. Test Contexts2. Wall How does sound propagate in this situation?

17. Test Contexts1a, 2a - Vacuum What happens without the medium (air or wall)?

18. Test questions - paraphrased • What is the mechanism of sound propagation in the air/wall? • How do particles of the medium vibrate, if at all, while the sound propagates? • How do particles of the medium travel,if at all, while the sound propagates? • What does this motion have to do with sound propagation – cause and effect relationship? • What does this motion have to do with sound propagation – time relationship? • What happens with sound propagation in thevacuum?

19. Displaying the test results • Several representations of students’ state of understanding • Available in real time and in post instruction analysis • Consistency: • Consistent – a student uses one model(Pure model state) • Inconsistent – a student uses more than one model(Mixed model state)

20. Using a particular model Pre Instruction; Calculus based; University; NY Inconsistently Consistently N = 100

21. Using a particular model at least once Pre Instruction; Calculus based; University; NY Inconsistently Consistently N = 100

22. Movements of particles of the medium Pre Instruction; Calculus based; University; NY (+) Random Travel (+) Travel Away From The source Vibration on the Spot N = 100

23. Model states Pre Instruction; Calculus based; University; NY Mixed Any Pure Other Mixed Entity Pure Wave Mixed Ear-Wave N = 100

24. Correctness Pre Instruction; Calculus based; University; NY N = 100

25. Using a particular model Pre Instruction; Calculus based; University; NY Inconsistently Consistently N = 100

26. Using a particular model Post Instruction; Calculus based; University; NY Inconsistently Consistently N = 95

27. Movements of particles of the medium Pre Instruction; Calculus based; University; NY (+) Random Travel (+) Travel Away From The source Vibration on the Spot N = 100

28. Movements of particles of the medium Post Instruction; Calculus based; University; NY (+) Random Travel (+) Travel Away From The source Vibration on the Spot N = 95

29. Correctness Pre Instruction; Calculus based; University; NY N = 100

30. Correctness Post Instruction; Calculus based; University; NY N = 95

31. Test validityBuilt and shown through: • Interviews with students • Expert reviews • Role playing validation with experts • Validity strengthening test development procedures • Tables of content and construct specifications • Meaningful correlations between all answer choices • Instructional sensitivity of the test • Stability (reliability) of results obtained in the large scale survey… • across different educational levels • across different institutions at equivalent educational levels • across different course levels at same institutions

32. Constructing the test Four steps of test construction and validation: • Pilot testing large open ended survey – settling on models, choosing contexts • Pre-survey testing expert validation, 7 choice survey (with none of the above, more than one of the above), correlation analysis of answer choices, refinement through interviews • Survey testing large scale survey –correlation analysis, comparisons between levels, pre-post results; interview validation • Post Survey testing moderately large scale survey, role playing, expert validation

33. Survey participants

34. Survey phase - Validity interviews • 17 x 4 probes in the interviewed sample. • The invalid display of a model would have occurred in 6 instances (out of 68). • 8.8% of the probes • 3 instances because of 5a (+ another 3 that did not cause invalid probe)

35. Correlation analysis of answer choices

36. Post-Survey Testing Expert review: • To validate post survey version • Few minor items improved • Surveying: • To determine correlations between response items and see if changes made the desired effect. • Problems fixed • Role playing validation: • To validate new test version in an additional way • Perfect score

37. Comparing model distributionDifferent educational levels

38. Comparing model distribution Grouped models; Different Educational Levels

39. Comparing model distributionDifferent course levels

40. Pre-Post instruction difference *Gain (G) = (post-test) – (pre-test) **Normalized gain (h) = gain / (maximum possible gain) (Hake, 1997).

41. Test packageProspective uses of test, test questions Online package related to test and analysis of data available at: http://web.phys.ksu.edu/role/sound/ Formative assessment combined with any instructional method/approach • “traditional” • “progressive” • “misconception oriented” • Model – cause • Misconception – symptom As peer instruction questions (not model defining) Not recommended as a summative assessment

42. Limitations • Common to multiple choice tests • Answer options do affect students understanding / models • Test taking strategies may obscure results • Test projects no model state as mixed model state and possibly pure model state.

43. Future researchUnique approach - Wide themes opened • Applicability of the approach in other domains of physics: • Is the approach “hybrid model-(in)dependent”? • Applicability in domains of other natural sciences? • How effectively teachers can implement the real-time aspect of this testing approach? • Instructional utility of this type of testing: Will addressing of the underlying models in real time help students learn? • Possibility of individualized addressing of student’s models in real time? • Applicability of the testing approach in eliciting non-cognitive psychological constructs: • Personality tests: Would it provide information that current tests in that field do not? • Reduction of items when compared to Likert scale

44. More Information / Feedback zhrepic@fhsu.edu www.fhsu.edu/~zhrepic(www.hrepic.com) Thank You!

45. Literature • Brown, D., & Clement, J. (1992). Clasroom teaching experiments in mechanics. In R. Duit, Goldberg, F. , Niedderer, H. (Ed.), Research in physics learning: Theoretical issues and empirical studies (pp. 380-389). Kiel: IPN. • Galili, I., Bendall, S., & Goldberg, F. M. (1993). The effects of prior knowledge and instruction on understanding image formation. Journal of Research in Science Teaching, 30(3), 271-301. • Hrepic, Z. (2002). Identifying students' mental models of sound propagation. Unpublished Master's thesis, Kansas State University, Manhattan. • Hrepic, Z., Zollman, D., & Rebello, S. (2002). Identifying students' models of sound propagation. Paper presented at the 2002 Physics Education Research Conference, Boise ID. • Itza-Ortiz, S. F., Rebello, S., & Zollman, D. A. (2004). Students’ models of Newton’s second law in mechanics and electromagnetism. European Journal of Physics, 25, 81–89. • Otero, V. K. (2001). The process of learning about static electricity and the role of the computer simulator. Unpublished Ph.D. Dissertation, University of California, San Diego, CA. • Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning & Instruction, 4, 45-69. • Greca, I. M., & Moreira, M. A. (2002). Mental, physical, and mathematical models in the teaching and learning of physics. Science Education, 86(1), 106-121. • diSessa, A. A. (2002). Why "conceptual ecology" is a good idea. In M. Limon & L. Mason (Eds.), Reconsidering conceptual change: Issues in theory and practice (pp. 29-60). Dordrecht, Netherlands: Kluwer Academic Publishers. • Hrepic, Z., Zollman, D., & Rebello, S. (2002). Identifying students' models of sound propagation. Paper presented at the 2002 Physics Education Research Conference, Boise ID. • Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning & Instruction, 4, 45-69. • Physics Education Group at Arizona State University. (2000). Modeling Instruction Program [www]. Arizona State University. Retrieved 24. Aug. 2003, 2003, from the World Wide Web: http://modeling.la.asu.edu/ • Clement, J. M. (2003). Re: testing to discriminate between students vs other approaches: PhysLrnR post of 18 Apr 2003 10:35:46 -0500; online at <http://listserv.boisestate.edu/cgi-bin/wa?A2=ind0304&L=physlrnr&F=&S=&X=412D9A0AB9B02985B9&Y=zhrepic@phys.ksu.edu&P=6582>. • Hanna, G. S. (1993). Better teaching through better measurement. Orlando, Florida: Harcourt Brace Jovanovic, Inc. • Oosterhof, A. (2001). Classroom applications of educational measurement. Upper Saddle River, New Jersey: Prentice Hall, Inc.