Experimental Methods for Studying Student Metacognition and Affect

1. Brett van de Sande Arizona State University Experimental Methods for Studying Student Metacognition and Affect PERC 2010 workshop during student problem-solving (homework)

2. Our group/team Also: Kasia Muldner, Winslo Burleson, Asael Sorenson, Raj Ranganathan Kurt.Vanlehn@asu.edu bvds@asu.edu University of Pittsburgh

3. Brett van de Sande Outline • Theoretical background • Definitions of Metacognition & Affect • Why study them? • Survey of methods, with examples; focus on Intelligent tutor systems • Hands on: try solving a problem with Andes • Possible future studies

4. Brett van de Sande Define Metacognition • Thinking about cognition (memory, reasoning, learning, etc.) itself. • It is “thinking about thinking.” • For learning, this includes: • Planning: the design of the learning process, choice of strategy • Monitoring: comparing actual progress to the desired one • Self assessment: the ability to correctly evaluate one's own knowledge level • Debugging: identifying sources of failure and overcoming those

5. Brett van de Sande Definitions: Affect • A feeling or emotion, as distinguished from cognition, thought, or action (conation) • In learning: boredom, confusion, delight, flow, frustration, and surprise • Time scales: • Long term (weeks/months): “I love/hate physics” • Short term (seconds): “What am I supposed to do for step 2?” • “Affective trajectory” (Michael Baker)

6. Brett van de Sande Why study metacognition?Why study affect? Why in the context of homework? • Groups of 3-5, discuss • Collect results: • Hw is less structured than class, teacher not direct influence; habits more important than topics; affect important for time decisions; motivation <-> affect. • Experts vs students: experts use m-cog skills, should be part of assessment. Sense of how student is framing activity: “get an answer” vs. “understand” mc skill students learn is something to learn & affects other learning. • Differentiating between “challenge” & “frustration” determine object of frustration. • Identify frustration & intervene (encourage).

7. Brett van de Sande Survey of Methods • Metacognition and Affect dissimilar • Experimental probes • Survey • Interview • Physiological (posture sensors) • Behavioral (decision-making) • Neurological (MRI, etc) new, $$$

8. Brett van de Sande Surveys • Example: CLASS survey (Colorado) 4. It is useful for me to do lots and lots of problems when learning physics. 9. I find that reading the text in detail is a good way for me to learn physics. 25. I enjoy solving physics problems. 34. I can usually figure out a way to solve physics problems. 40. If I get stuck on a physics problem, there is no chance I'll figure it out on my own.

9. Brett van de Sande • Study: “Are Students’ Responses and Behaviors Consistent?”(U. Wutchana, N. Emarat and E. Etkina, 2009) • From Summary: • “…the observations indicate that students behave differently from what they report on the survey. Thus it might be difficult to access students’ beliefs about physics and learning physics by using surveys only.”

10. Brett van de Sande Surveys • Good: • Easy to do • Probes mental state, “what’s going on inside” • Bad: • Longer time scales, limited number of data points • Hard to scale (what does a “3” mean for an emotion?) • Susceptible to bias (give answer that is perceived to be desired by the instructor). • Meta-metacognition, Meta-Affect: student reflection on … • “After the fact:” the metacognitive activity or event that caused the affective state has come and gone.

11. Brett van de Sande Interview techniques • Schoenfeld (1992) advocates study of metacognition. • At the middle school level, Lester, Garofalo & Kroll (1989, June) investigation “designed to study the role of metacognition (i.e. the knowledge and control of cognition) in seventh graders’ mathematical problem solving.” Assessment data and tools employed before, during, and after the instruction included written tests, clinical interviews, observations of individual and pair problem-solving sessions, and videotapes of the classroom instruction.

12. Brett van de Sande Interview techniques • Good: • In depth probe of metacognitive processes • Can adapt to student response • Bad: • Time consuming (small sample size) • Analysis hard (small sample size) • Affect susceptible to student bias (student reports what they believe experimenter wants) • Presupposes a theoretical framework (bias in interview questions)

13. Brett van de Sande Physiological measures • Probes: • Facial expression • Skin resistance • Heart rate • Butt sensor • Example: Art Grasser & AutoTutor AutoTutor is natural language tutor for qualitative Newtonian physics. experiments where students are monitored for facial expressions and butt sensor as they solve problems with tutor. Classify affective states: boredom, confusion, delight, flow, frustration, & surprise

14. Brett van de Sande Physiological measures • Good: • Hard to fake responses (authentic) • Real time monitoring • Bad: • Intrusive, restricted to lab setting • Expensive equipment • Video methods need manual analysis • Still need to connect to actual Affective state via survey or interview methods.

15. Brett van de Sande Behavioral Measures • “Decision making in an authentic context” • Does the student drop the class/change major? • Do they ask for help or try again without help? • Read a hint or immediately ask for the next hint? • Good: • Easy to measure (in context of tutor system) • In many cases, measure often, on short time scale • Authenticity possible • Unobtrusive • Bad: • Need to connection to student mental state

16. Brett van de Sande Example behavioral measure: Gaming studies • Gaming (Baker): “Attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material.” • Examples: • Guessing & looking at tutor response • Drilling through hint sequences • Follows from affective states of boredom & confusion (Rodrigo et al, 2007) • Metacognitive strategy • Student uses gaming to quickly get through previously mastered, but tedious material. • Compensate for errors/weaknesses in the tutoring system.

17. Brett van de Sande Session as dialog between tutor and student Use timing of student action after the tutor responds to student’s previous step: Example: “An Analysis of Gaming Behaviors in an Intelligent Tutoring System” (K. Muldner, et al, 2010) Student Action Tutor Response

18. Brett van de Sande Try the new version of Andes • Use example problem to explain instructional strategy • Features instructors care about • Try it out!

19. Brett van de Sande Andes from the student’s perspective:an example from statics Students at the USNA took an average of 15 minutes (median 13 minutes) to solve this.

21. Upon request, Andes gives hints for what to do next Define variables Draw free body diagram (3 vectors and body) Define coordinates (3 choices for this problem)

22. Immediate Red/green feedback for student actions Principles-based help for Incorrect entry

23. Students must provide units

24. score Students work with algebraic expressions Andes calculates numerical answer

25. Brett van de Sande Instructor’s perspective • Homework component of a standard two-semester introductory course (with lecture and textbook) • Works in any web browser • Over 500 problems • 100% coverage of USNA fall course (no thermodynamics) • 90% of spring course (no calculus or modern physics) • Not tied to a specific textbook (have used Knight, Giancoli, & Serway). • Student testing this summer, classroom beta testing this fall (October). • Offered through: Open Learning Initiative (OLI), WebAssign, & LON-CAPA (thanks to Andy Pawl).

26. Brett van de Sande Try it out • http://andestutor.org/sets • Start with problem vec1ay and follow the instructions in the tutor pane. • If you are stuck or unsure why an entry is incorrect, you can ask the Andes Tutor for Help. Just click on the help button at the bottom of the Tutor pane. • If you have a comment or complaint, type it in the text box at the bottom of the Tutor pane. These are logged and will be read by us later. • When you have completed a problem, click on the submit button to exit.

27. Brett van de Sande Possible future studies • Probes: • Log student actions (solution steps, errors, help requests), as well as tutor interpretation, with time. • Could have fine-grained logging: keystrokes, mouse clicks, menu usage. • Can dialog with students in tutor pane before/during/after the problem solving process. • Experimental contexts (each has pros and cons): • Student in dorm room/home. • In class (can have working with pairs, microphones) • Laboratory (can also have video, physiological sensors).

28. Brett van de Sande Future experiments • Theoretical question • metacognition and/or affect? • Time scale of “affective trajectory” • Verbal and behavioral aspect? • Experimental design • Probe & experimental context • Pre-test/post-test • Control & experimental conditions

29. Thank you. bvds@asu.edu Kurt.Vanlehn@asu.edu