Developing a Useful Instrument to Assess Student Problem Solving

1. Developing a Useful Instrument to Assess Student Problem Solving Jennifer L. Docktor Ken Heller Physics Education Research & Development Group http://groups.physics.umn.edu/physed DUE-0715615

2. Problem Solving Measure • Problem solving is an important part of learning physics. • Unfortunately, there is no standard way to measure problem solving so that student progress can be assessed. • The goal is to develop a robust instrument to assess students’ written solutions to physics problems, and obtain evidence for reliability, validity, and utility of scores. • The instrument should be general • not specific to instructor practices or techniques • applicable to a range of problem topics and types Jennifer Docktor, University of Minnesota

3. Reliability, Validity, & Utility • Reliability – score agreement • Validity evidence from multiple sources • Content • Response processes • Internal & external structure • Generalizability • Consequences of testing • Utility - usefulness of scores AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741-749. Jennifer Docktor, University of Minnesota

4. Overview of Study • Drafting the instrument (rubric) • Preliminary tests with two raters (final exams and instructor solutions) • Training exercise with graduate students • Analysis of tests from an introductory mechanics course • Student problem-solving interviews (in progress) Jennifer Docktor, University of Minnesota

5. What is problem solving? • “Problem solving is the process of moving toward a goal when the path to that goal is uncertain” (Martinez, 1998, p. 605) • What is a problem for one person might not be a problem for another person. • Problem solving involves decision-making. • If the steps to reach a solution are immediately known, this is an exercise for the solver. Martinez, M. E. (1998). What is Problem Solving? Phi Delta Kappan, 79, 605-609. Hayes, J.R. (1989). The complete problem solver (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Schoenfeld, A.H. (1985). Mathematical problem solving. Orlando, FL: Academic Press, Inc. Jennifer Docktor, University of Minnesota

6. Problem Solving Process • Organize problem information • Introduce symbolic notation • Identify key concepts Understand / Describe the Problem • Use concepts to relate target to known information Devise a Plan • appropriate math procedures Carry Out the Plan Look Back • check answer Pόlya, G. (1957). How to solve it (2nd ed.). Princeton, NJ: Princeton University Press. Reif, F. & Heller, J.I. (1982). Knowledge structure and problem solving in physics. Educational Psychologist, 17(2), 102-127. Jennifer Docktor, University of Minnesota

7. Problem Solver Characteristics Experienced solvers: • Hierarchical knowledge organization or chunks • Low-detail overview / description of the problem before equations • qualitative analysis • Principle-based approaches • Solve in symbols first • Monitor progress, evaluate the solution Inexperienced solvers: • Knowledge disconnected • Little representation (jump to equations) • Inefficient approaches (formula-seeking & solution pattern matching) • Early number crunching • Do not evaluate solution Chi, M. T., Feltovich, P. J., & Glaser, R. (1980). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152. Larkin, J., McDermott, J., Simon, D.P., & Simon, H.A. (1980). Expert and novice performance in solving physics problems. Science, 208(4450), 1335-1342. Jennifer Docktor, University of Minnesota

8. Instrument at a glance (Rubric) SCORE CATEGORY: (based on literature) Useful Description Physics Approach Specific Application Math Procedures Logical Progression • Minimum number of categories that include relevant aspects of problem solving • Minimum number of scores that give enough information to improve instruction Want Jennifer Docktor, University of Minnesota

9. Rubric Category Descriptions • Useful Description • organize information from the problem statement symbolically, visually, and/or in writing. • Physics Approach • select appropriate physics concepts and principles • Specific Application of Physics • apply physics approach to the specific conditions in problem • Mathematical Procedures • follow appropriate & correct math rules/procedures • Logical Progression • (overall) solution progresses logically; it is coherent, focused toward a goal, and consistent Jennifer Docktor, University of Minnesota

10. Rubric Scores (in general) NOT APPLICABLE (NA): Jennifer Docktor, University of Minnesota

11. Jennifer Docktor, University of Minnesota

12. Early Tests of the Rubric • Preliminary testing (two raters) • Distinguishes instructor & student solutions • Score agreement between two raters – good • Training Exercise (8 Graduate Students) • Half scored a mechanics problem, half E&M • Scored 8 student solutions with the rubric, received example scores & rationale for first 3, then re-scored 5 and scored 5 new solutions • Answered survey questions about the rubric Jennifer Docktor, University of Minnesota

13. Grad Student Comments • Influenced by traditional grading experiences • Unwilling to score math & logic if physics incorrect • Desire to weight categories “I don't think credit should be given for a clear, focused, consistent solution with correct math that uses a totally wrong physics approach” (GS#1) “[The student] didn't do any math that was wrong, but it seems like too many points for such simple math…I would weigh the points for math depending on how difficult it was. In this problem the math was very simple” (GS#8) Jennifer Docktor, University of Minnesota

14. Grad Student Comments • Difficulty distinguishing categories • Physics approach & application • Description & logical progression “Specific application of physics was most difficult. I find this difficult to untangle from physics approach. Also, how should I score it when the approach is wrong?” (GS#1) “I think description & organization are in some respect very correlated, & could perhaps be combined” (GS#5) Jennifer Docktor, University of Minnesota

15. Inter-rater Agreement Fair agreement Moderate agreement *Weighted kappa difference significant at 0.01 level Jennifer Docktor, University of Minnesota

16. Written Training Exercise • Minimal written training was insufficient • Confusion about NA scores (want more examples) • Score agreement improved significantly with training, but is not yet at an optimal level • Difficulty distinguishing physics approach & application • Math & Logical progression most affected by training • Multi-part problems more difficult to score • Grad students influenced by traditional grading experience Jennifer Docktor, University of Minnesota

17. Analysis of Tests • Calculus-based introductory physics course for Science & Engineering students (mechanics) • Fall >900 students split into 4 lecture sections • 4 Tests during the semester • 5 MC, 2 individual problems, 1 group problem • Problems graded in the usual way by teaching assistants • After they were graded, I used the rubric to evaluate 8 individual problems spaced throughout the semester • Approximately 300 student solutions per problem (copies made by TAs from 2 sections) Jennifer Docktor, University of Minnesota

18. Exam 3 Question 1 Show all work! The system of three blocks shown is released from rest. The connecting strings are massless, the pulleys ideal and massless, and there is no friction between the 3 kg block and the table. (A) At the instant M3is moving at speed v, how far (d) has it moved from the point where it was released from rest? (answer in terms of M1, M2, M3, g and v.) [10 points] (B) At the instant the 3 kg block is moving with a speed of 0.8 m/s, how far, d, has it moved from the point where it was released from rest? [5 pts] (C)…. (D)…. SYMBOLIC CUES ON MASS 3 How would you solve part A? Jennifer Docktor, University of Minnesota

19. Grader Scores Excludes part c) multiple choice question. Average score the same (9 points or ~ half). Jennifer Docktor, University of Minnesota

20. Rubric Scores • Useful Description: Free-body diagram (not necessary for energy approach) • Physics Approach: Deciding to use Newton’s 2nd Law or Energy Conservation • Specific Application: Correctly using Newton’s 2nd Law or Energy Cons. • Math Procedures: solving for target • Logical Progression: clear, focused, consistent Jennifer Docktor, University of Minnesota

21. Common Responses Statements in red suggest students focused on M3, which was cued in the problem statement Jennifer Docktor, University of Minnesota

22. Example Student Solution Jennifer Docktor, University of Minnesota

23. Example Student Solution Only consider kinetic energy of mass M3. ? Was cued in problem statement. Jennifer Docktor, University of Minnesota

24. Example Student Solution • (E1=E2=E3) Jennifer Docktor, University of Minnesota

25. Example Student Solutions Considers forces on M3, and uses T=mg (incorrect) Jennifer Docktor, University of Minnesota

26. Example Student Solution Answer is correct, but reasoning for “F” unclear Jennifer Docktor, University of Minnesota

27. Findings • The rubric indicates areas of student difficulty for a given problem • i.e. the most common difficulty is specific application of physics whereas other categories are adequate • Focus instruction to coach physics, math, clear and logical reasoning processes, etc. • The rubric responds to different problem features • For example, in this problem visualization skills were not generally measured. • Modify problems to elicit / practice processes Jennifer Docktor, University of Minnesota

28. Problem Characteristics that could Bias Problem Solving Description: • Picture given • Familiarity of context • Prompts symbols for quantities • Prompt procedures (i.e. Draw a FBD) Physics: • Prompts physics • Cue focuses on a specific objects Math: • Symbolic vs. numeric question • Mathematics too simple (i.e. one-step problem) • Excessively lengthy or detailed math Jennifer Docktor, University of Minnesota

29. Summary • A rubric has been developed from descriptions of problem solving process, expert-novice research studies, and past studies at UMN • Focus on written solutions to physics problems • Training revised to improve score agreement • Rubric provides useful information that can be used for research & instruction • Rubric works for standard range ofphysics topics in an introductory mechanics course • There are some problem characteristics that make score interpretation difficult • Interviews will provide information about response processes Jennifer Docktor, University of Minnesota

30. docktor@physics.umn.edu http://groups.physics.umn.edu/physed Additional Slides (if time permits)

31. Exam 2 Question (Different) A block of mass m = 3 kg and a block of unknown mass M are connected by a massless rope over a frictionless pulley, as shown below. The kinetic frictional coefficient between the block m and the inclined plane is μk= 0.17. The plane makes an angle 30owith horizontal. The acceleration, a, of the block M is 1 m/s2 downward. (A) Draw free-body diagrams for both masses. [5 points] (B) Find the tension in the rope. [5 points] (C) If the block M drops by 0.5 m, how much work, W, is done on the block m by the tension in the rope? [15 points] A block of known mass m and a block of unknown mass M are connected by a massless rope over a frictionless pulley, as shown. The kinetic frictional coefficient between the block m and the inclined plane is μk. The acceleration, a, of the block M points downward. (A) If the block M drops by a distance h, how much work, W, is done on the block m by the tension in the rope? Answer in terms of known quantities. [15 points] NUMERIC SYMBOLIC Jennifer Docktor, University of Minnesota

32. Grader Scores AVERAGE: 15 points Numeric, prompted: Several people received the full number of points, some about half. AVERAGE: 16 points Symbolic: Fewer students could follow through to get the correct answer. Jennifer Docktor, University of Minnesota

33. Rubric Scores prompted • Useful Description: Free-body diagram • Physics Approach: Deciding to use Newton’s 2nd Law • Specific Application: Correctly using Newton’s 2nd Law • Math Procedures: solving for target • Logical Progression: clear, focused, consistent Jennifer Docktor, University of Minnesota

34. Solution Examples • (numeric question w/FBD prompted) Could draw FBD, but didn’t seem to use it to solve the problem Jennifer Docktor, University of Minnesota

35. Solution Example • (numeric question w/FBD prompted) NUMBERS NOTE: received full credit from the grader Jennifer Docktor, University of Minnesota

36. (numeric question w/FBD prompted) Jennifer Docktor, University of Minnesota

37. Symbolic form of question Jennifer Docktor, University of Minnesota

38. Symbolic form of question Left answer in terms of unknown mass “M” Jennifer Docktor, University of Minnesota

39. Findings about the Problem Statement • Both questions exhibited similar problem solving characteristics shown by the rubric. However • prompting appears to mask a student’s inclination to draw a free-body diagram • the symbolic problem statement might interfere with the student’s ability to construct a logical path to a solution • the numerical problem statement might interfere with the student’s ability to correctly apply Newton’s second law • In addition, the numerical problem statement causes students to manipulate numbers rather than symbols Jennifer Docktor, University of Minnesota

40. Findings about the Rubric • The rubric provides significantly more information than grading that can be used for coaching students • Focus instruction on physics, math, clear and logical reasoning processes, etc. • The rubric provides instructors information about how the problem statement affects students’ problem solving performance • Could be used to modify problems Jennifer Docktor, University of Minnesota

41. References http://groups.physics.umn.edu/physed docktor@physics.umn.edu P. Heller, R. Keith, and S. Anderson, “Teaching problem solving through cooperative grouping. Part 1: Group versus individual problem solving,” Am. J. Phys., 60(7), 627-636 (1992). J.M. Blue, Sex differences in physics learning and evaluations in an introductory course. Unpublished doctoral dissertation, University of Minnesota, Twin Cities (1997). T. Foster, The development of students' problem-solving skills from instruction emphasizing qualitative problem-solving. Unpublished doctoral dissertation, University of Minnesota, Twin Cities (2000). J.H. Larkin, J. McDermott, D.P. Simon, and H.A. Simon, “Expert and novice performance in solving physics problems,” Science 208 (4450), 1335-1342. F. Reif and J.I. Heller, “Knowledge structure and problem solving in physics,” Educational Psychologist, 17(2), 102-127 (1982). Jennifer Docktor, University of Minnesota

42. Additional Slides

43. Independent scoring of student solutions by a PER graduate student and a high school physics teacher (N=160) Inter-rater Reliability Kappa: <0 No agreement 0-0.19 Poor 0.20-0.39 Fair 0.40-0.59 Moderate 0.60-0.79 Substantial 0.80-1 Almost perfect Jennifer Docktor, University of Minnesota

44. Inter-rater Agreement Fair agreement Moderate agreement *Weighted kappa difference significant at 0.01 level Jennifer Docktor, University of Minnesota

45. All Training in Writing: Example Training includes the actual student solution CATEGORY RATIONALE SCORE Jennifer Docktor, University of Minnesota

46. Ratings Before & After Training Math & Logic most affected by training Jennifer Docktor, University of Minnesota

47. Kappa Jennifer Docktor, University of Minnesota

48. Weighted Kappa Jennifer Docktor, University of Minnesota

49. Exam 1 Question 1 • A block of mass m=2.5 kg starts from rest and slides down a frictionless ramp that makes an angle of θ=25o with respect to the horizontal floor. The block slides a distance d down the ramp to reach the bottom. At the bottom of the ramp, the speed of the block is measured to be v=12 m/s. • Draw a diagram, labeling θ and d. [5 points] • b) What is the acceleration of the block, in terms of g? [5 points] • c) What is the distance, d, in meters? [15 points] INSTRUCTOR SOLUTION Jennifer Docktor, University of Minnesota

50. Grader Scores >40% of students received the full points on this question Was this an exercise or a problem? Jennifer Docktor, University of Minnesota