What is mathematical understanding? How can we measure it? Theoretical and empirical reflections

1. What is mathematical understanding? How can we measure it? Theoretical and empirical reflections Dr. Jon STAR Harvard University Seminar Presentation, Department of Elementary Education Seminar, Middle East Technical University, Ankara, Turkey; 27 August 2007

2. Acknowledgements • Dr. Cigdem HASER • Department of Elementary Education • METU

3. Some background information • Who is Dr. Jon STAR? • What is his connection to METU? • What is the format of this seminar? • Who is the intended audience for this seminar? • Can I interrupt the seminar with questions?

4. My goals for the seminar • Push you to think more carefully about • What “understanding” means • How understanding is assessed • Convince you (if you need convincing!) that the nature of and assessment of understanding are issues that you need to think seriously about in your work with teachers and students

5. Getting started • Please get in groups of 2-3 people and share your answers to the following questions: • What does it mean to understand a mathematical topic? • How can you tell when someone has developed understanding of a mathematical topic?

6. “Understanding” words? • understand, know, explain, concept, procedure, rote, memorize, transfer, apply, recall, generate, prove, problem solving, network, connections, web, schema, ...

7. Does this assess understanding?

8. Does this assess understanding?

9. Does this assess understanding?

10. Does this assess understanding?

11. Does this assess understanding?

12. Does this assess understanding?

13. Generalizing? • Can you make any general statements about what understanding is and how to assess it? • Five categories of responses to the question • “What does it mean for someone to understand a mathematical topic”?” • And my thoughts on each...

14. What is understanding? 1. Understanding is hard to define. • I know it when I see it, but I can’t define it • Understanding is difficult to define precisely • People smarter than I am have tried and failed over the past several thousand years! • Is it really necessary to come up with a good definition of “understanding”?

15. 1. Understanding is hard to define. Jon says... • YES, it is necessary to come up with a definition • Understanding is often the instructional outcome that we aim for. • We need a target when we are designing instructional materials • Need to measure whether our goal has been achieved • Designing measures requires a careful and precise definition and operationalization of understanding

16. What is understanding? 2. Cite an expert. • You can’t expect me to re-invent the wheel! • I use understanding in the same way that (insert your favorite mathematics education scholar here) does. • For example, Hiebert, Skemp, Brownell, Sfard, Star? • If another framework is widely used and accepted, isn’t it OK to just use it?

17. 2. Cite an expert. Jon says... • Certainly it is important to ground your ideas about understanding with the established literature • It helps to connect with a literature that is well-known and a framework for understanding that is commonly used • But...

18. 2. Cite an expert. Jon says... • Most who “piggy-back” on others’ frameworks are not particularly knowledgeable about who and what they are citing • Pervasive citation of Hiebert (1986) and Skemp (1978) fall into this category • Before citing someone’s ideas about understanding, become very familiar with the framework

19. 2. Cite an expert. Jon says... • Many assume that different frameworks mean the same thing • For example, that procedural/conceptual (Hiebert) is the same as relational/instrumental (Skemp) • Different frameworks and terminological distinctions shed light on different aspects of understanding • All theoretical frameworks are not the same

20. 2. Cite an expert. Assessment frameworks • Many adopt a framework because of its use in national or international assessments • Widespread use of an assessment framework is not a compelling reason to choose a framework • Assessment frameworks are idiosyncratic and difficult to faithfully apply outside of the assessment • Assessment frameworks often not theoretically sound

21. 2. Cite an expert. Consider these frameworks • NAEP (US national maths assessment) • Procedural knowledge, Conceptual understanding, Problem solving • PISA (large international maths assessment) • Reproduction, Connections, Reflection • TIMSS (large international maths assessment) • Solving routine problems, Knowing facts and procedures, Reasoning, Using concepts

22. TIMSS 2003 8th grade M012042 Solving routine problemsKnowing facts and proceduresReasoningUsing concepts Knowing facts and procedures

23. TIMSS 2003 8th grade M022002 Solving routine problemsKnowing facts and proceduresReasoningUsing concepts Solving routine problems

24. TIMSS 2003 8th grade M022251 Solving routine problemsKnowing facts and proceduresReasoningUsing concepts Using concepts

25. TIMSS 2003 8th grade M022008 Solving routine problemsKnowing facts and proceduresReasoningUsing concepts Reasoning

26. NAEP 8th grade 2003-8M6 #27 Conceptual understandingProcedural knowledgeProblem solving Conceptual understanding

27. NAEP 8th grade 2003-8M7 #18 Conceptual understandingProcedural knowledgeProblem solving Procedural knowledge

28. NAEP 8th grade 2003-8M7 #11 Conceptual understandingProcedural knowledgeProblem solving Problem solving

29. 2. Cite an expert. Jon says... • Know what you are getting into when you cite an expert! • Don’t assume that assessment frameworks reflect a coherent and theoretically sound vision of what understanding is

30. What is understanding? 3. Knowledge organization in head. • Understanding refers to how knowledge is organized in someone’s memory/brain/head. • Understanding is when knowledge is tightly connected, is a web, is a network, is organized into schema, etc.

31. 3. Knowledge organization in head. Jon says... • It may be true that understanding is indicated by or a result of a certain organization of knowledge in the head • It may be helpful to use metaphors and abstractions (such as links and connections) to talk about knowledge organization • But...

32. 3. Knowledge organization in head. Jon says... • Defining understanding this way is not particularly helpful in designing assessments • The metaphors or abstractions are not directly transferable to learning and assessment • If links = understanding, do we teach links? • Does the ability to draw a concept map (with lots of links in a network) indicate understanding?

33. What is understanding? 4. Quality of verbalization. • A student understands if he/she can explain it to someone else. • If the student knows it well enough to provide a thorough and principled explanation to someone else, I am satisfied that he/she understands.

34. 4. Quality of verbalization. Jon says... • Certainly the ability to verbally explain is a potential indicator that a student understands • But...

35. 4. Quality of verbalization. Jon says... • Impractical in terms of assessment; we can’t interview every child to see if he/she understands • We need a more efficient way to tap understanding • Students’ verbalizations are often spontaneous • Typically not a well-thought-out, carefully worded, articulate response • Requires an interviewer to carefully probe and explore the student’s ZPD

36. What is understanding? 5. Quality of written performance. • Understanding is evident in what students do • When given a task/problem, we judge whether or not the student understands based on his/her performance • Verbalization can give an additional window into what students are thinking, and knowledge organization is potentially useful as a metaphor • “Intelligent performances” (Ryle)

37. 5. Quality of written performance. Jon says... • Closest to what I believe • Major implications for assessment • It is incumbent on me to design very good questions so that how students respond does indicate that they understand • Let’s talk about creating good assessments of understanding!

38. Assessments of understanding • Don’t be swayed by existing frameworks for categorizing items or by the assignment of particular items to categories • Questions from other assessments may be quite useful, but the category label (“Solving routine problems”) is likely not helpful

39. Avoid stereotypes • Stereotyped views of conceptual knowledge • “If it is possible to complete this question using a memorized procedure, then it does not assess understanding” • “If it is a multiple choice question, it cannot assess conceptual understanding”

40. Avoid stereotypes • Stereotyped views of procedural knowledge • Procedures are either known (and student can execute them, often by rote) or they are not known • Also consider items that assess “deep procedural knowledge” • knowledge of multiple procedures, knowledge of which procedures are better for certain circumstances, ability to adapt procedures to changing circumstances, ability to evaluate procedures See Star, 2005, 2007, articles in J. for Research in Mathematics Education for more on this.

41. Avoid “problem solving” • Many assessment frameworks include a category called “problem solving” • Overlaps with other categories (e.g., procedural knowledge and conceptual knowledge) • Word problems fall into multiple categories, not just problem solving • Has become a political term and thus relatively useless for mathematics educators

42. Name the concept • When designing assessments of conceptual knowledge, it is important to be able to name the target concept • Conceptual knowledge as knowledge of a concept • Rather than as a descriptor for a quality of knowledge

43. Validity • Evaluate the psychometic validity of your assessments • Cronbach’s alpha for inter-item reliability • Factor analyses to establish or confirm ‘grouping’ of items

44. Bottom line • Be thoughtful and deliberate about the frameworks, terms, and citations that you use • Make sure your definition of understanding is tightly linked to the ways that you operationalize and assess it

45. Examples from my research • I assess three components of understanding (of algebra equation solving) • Conceptual knowledge • Procedural knowledge • Procedural flexibility

46. Conceptual knowledge • Knowledge of key concepts used in equation solving • Concept of equivalence • Concept of variable

47. For example • If m is a positive number, which of these is equivalent to (the same as) m + m + m + m? (Responses are: 4m; m4; 4(m + 1); m + 4) • Concept of variable • For the two equations:213x + 476 = 984 213x + 476 + 4 = 984 + 4 Without solving either equation, what can you say about the answers to these equations? Explain your answer. • Concept of equivalence

48. Procedural knowledge • Knowledge of procedures • Ability to successfully execute equation solving procedures on problems that are similar to those seen in the instructional intervention (“familiar” or “learning” problems) • Ability to successfully execute equation solving procedures that are somewhat different than those seen in the intervention (“transfer” problems)

49. Procedural flexibility • Ability to generate, recognize, and evaluate multiple solution methods for the same problem • One example of “deep procedural knowledge” • Short-answer and multiple choice assessment designed to measure flexibility (e.g., Beishuizen, van Putten, & van Mulken, 1997; Blöte, Klein, & Beishuizen, 2000; Blöte, Van der Burg, & Klein, 2001; Star, 2005, 2007; Star & Seifert, 2006; Rittle-Johnson & Star, 2007)

50. Examples of flexibility items • A student’s first step for solving the equation 3(x + 2) = 12 was x + 2 = 4. What step did the student use to get from the first step to the second step? Do you think that this way of starting this problem is (a) a very good way; (b) OK to do, but not a very good way; (c) Not OK to do? Explain your reasoning. • For the equation 2(x + 1) + 4 = 12, identify all possible steps (among 4 given choices) that could be done next. • Solve 4(x + 2) = 12 in two different ways.