False Dichotomies: Are Conceptual Questions Easier than Numeric Questions?

1. False Dichotomies: Are Conceptual Questions Easier than Numeric Questions? Faculty Research & Creativity Forum October 30, 2008 D. MacIsaac, Physics J. Zawicki, Earth Sciences and Science Education L. Gomez, Physics SUNY Buffalo State College T. Johnson, Erie 1 BOCES, WNYRIC

2. Curriculum Standards • Frameworks • Syllabi • Guides • Blueprints • Benchmarks validity correlation • Objective tests • Performance assessments • Portfolios • Teacher Observations • Group Activities • Program Evaluations • Instructional styles • Print materials • Equipment • Facilities • Technology • Community alignment Assessment/Evaluation System Instructional Program False Dichotomies

3. Types of Analysis • Conventional • Difficulty (Facility) • Discrimination • Response pattern • Rasch • Item format • Difficulties analyzed in the context of issues:1 • Student • Instructional (Teacher, School) • Testing 1NYS Biology Mentor Network False Dichotomies

4. Student Instruction Validity Testing Test Validity • You didn’t teach the associated core major understandings. • You didn’t reinforce the core understandings enough. • You taught the core content wrong • Content Knowledge? • Literacy/Reading Comprehension? • Question interpretation Skills? • Misconception? • From previous instruction? • From culture contexts? • Insufficient reinforcement? • Effort? • Difficulty (Facility) Level? • Discrimination? • Placement on exam? • Visual distraction by nearby (graphic) items? • Style of Question? • Flawed item? False Dichotomies

5. Claim #1 - Newton’s Third Law Students do not “see” action/reaction pairs. Modified Benjamin Bloom: knowing using integrating False Dichotomies

6. Gravitational interaction 10 Earth’s mass is approximately 81 times the mass of the Moon. If Earth exerts a gravitational force of magnitude F on the Moon, the magnitude of the gravitational force of the Moon on Earth is: (1) F (3) 9F (2) F/81 (4) 81F False Dichotomies

7. Claim #2 - Mass/Weight • Students have a difficult time differentiating between mass and weight. False Dichotomies

8. Mass/Weight A 2.00 kilogram object weighs 19.6-newtons on Earth. If the acceleration due to gravity on Mars is 3.71 meters per second2, what is the object’s mass on Mars? 1) 2.64 Kg 2) 2.00 Kg 3) 19.6 N 4) 7.42 N False Dichotomies

9. Mass/Weight A horizontal force of 8.0 newtons is used to pull a 20.-newton wooden box moving toward the right along a horizontal, wood surface, as shown. 61. Starting at point P on the diagram in your answer booklet, use a metric ruler and a scale of 1.0 cm = 4.0 N to draw a vector representing the normal force acting on the box. Label the vector FN. [1] D = 0.49 62. Calculate the magnitude of the fricitional force acting on the box. (Show all work.) [2] D = 0.72 63. Determine the magnitude of the net force acting on the box. [1] D = 0. 47 64. Determine the mass of the box. [1] D = 0.76 65. Calculate the magnitude of the acceleration of the box. [Show all work.] [2] D = 0.80 False Dichotomies

10. Claim #3 - Inverse Square Law • Sample items 2004-2007 • Range of item difficulties… False Dichotomies

11. June 2004 (D=0.59) False Dichotomies

12. June 2007 (D=0.78) False Dichotomies

13. June 2004 (D=0.67) False Dichotomies

14. June 2005 (D=0.67) False Dichotomies

15. June 2006 (D=0.82) False Dichotomies

16. June 2005 (D=0.41) False Dichotomies

17. Conclusions Teachers often underestimate the critical role played by conceptual physics reasoning which is quite difficult to teach and learn, and focus exclusively on numeric problem solving which is often already taught to students in other classes – like in chemistry and mathematics courses. The conceptual questions on NYSED Regents’ Physics exams are routinely amongst the most difficult items for HS physics students, and a common reaction from students and less successful teachers is that these are “trick questions.” However, these conceptual questions lie at the heart of the physics, and calculating physical quantities that cannot be reliably interpreted by students shows both a weakness in physics teaching and physics learning. Calculation emerges from, is validated and interpreted by conceptual understanding, and conceptual learning is not useful in physics unless it can be used to numerically predict natural phenomena. • Additional information: J. Zawicki <zawickjl@buffalostate.edu> False Dichotomies

18. We have observed amongst pre-service and new HS Physics Teachers a commonbelief that physics problems that do not involve mathematical calculations (aka “conceptual problems”) are easier than those involving mathematical calculations (aka “numeric problems”). Teachers are very likely to allocate student instruction and learning resources in physics according upon their beliefs. NY HS Physics teachers are driven by the NYSED Regents’ Physics Exam. Hence, we have selected and analyzed several elementary mechanics questions from about 1500 student scores on various NYSED Regents Physics Exams that challenge teacher perceptions of difficulty and numeric vs conceptual content. False Dichotomies

19. Example 1: Momentum, inertia and mass. The June 2003 Regent’s Physics exams juxtaposed several numeric and conceptual problems including (on the same page) items #12 and #15.   Item #15, a conceptual problem requiring simple definition (of inertia or mass, part of the definition of momentum) that presented extraneous numeric data was later determined to be the most difficult item on that exam offering – only about thirty eight percent of about a thousand students were able to answer it correctly. Item #12, a standard “cookbook-like” problem applying these same definitions and principles (calculating and conserving momentum in a collision) via multiple simple calculations turned out to be in the easiest third of the exam that year, answered correctly by over seventy percent of the students that year. False Dichotomies

20. Example 2: Mass and weight. A significant fraction of the NYSED Regents physics exam items make use of mass and weight calculations. Students have asked to calculate weight (a force) from mass and apply this information in different problems on every recent exam, often in complex multi-step problems as shown in the example. Students do relatively well on these items. Periodically the exam directly challenges the student to differentiate between mass and weight. These items are usually the most difficult items on that offering of the exam. False Dichotomies

21. Example 3: Gravitation interactions. NYSED Regents Physics exams prevalently include calculation of inverse square law forces (electrostatic and gravitation), often combined with proportional reasoning and often requiring interpretations of graphic representations of these forces.   When presented as a direct calculational exercise as seen on item 14 from the June 2007 exam, students perform well, despite the fact that this calculation requires significant expertise with scientific notation and calculators.   When asked for simple restatements of these gravitational interactions (the definition of any force), students find these questions about twice as difficult, amongst the most difficult on the exam. See item 12 from the June 2005 exam. False Dichotomies