Windows Into The Mind

1. Windows Into The Mind Richard J. Shavelson & Maria Araceli Ruiz-Primo Stanford University Invited Talk University of Ancona Ancona, Italy June 26, 2000

2. Assessing Knowledge Structure DENNIS THE MENACE “GEE! I’M GLAD WE DID CONCEPT MAPS - NOW WE BOTH KNOW WHAT I LEARNED TODAY!!! ” Shavelson & Ruiz-Primo: Windows Into The Mind

3. Overview • Cognitive theory underlying assessments of knowledge structure • Direct and indirect methods of assessing “cognitive structure” • Direct method: Concept maps • Indirect method: Cognitive maps • Link between concept and cognitive maps • Concluding comments Shavelson & Ruiz-Primo: Windows Into The Mind

4. Toward An Achievement Framework:Knowledge Types and Dimensions Declarative Procedural Strategic Knowledge Knowledge Knowledge (Knowing the “that”) (Knowing the “how”) (Knowing the “which,” “when,” and “why”) Proficiency Low High Extent (How much?) Structure (How is it organized?) Others (Precision? Efficiency? Automaticity?) Domain-specific content: • facts • concepts • principles Domain-specific production systems Problem schemata/ strategies/ operation systems Cognitive Tools: Planning Monitoring Shavelson & Ruiz-Primo: Windows Into The Mind

5. Associative and Semantic Models of Memory • Nodes represent concepts • Lines represent relations among concepts • Probe memory by asking about concepts (or lines) is a contains is an example of Associative Semantic Shavelson & Ruiz-Primo: Windows Into The Mind

6. Concepts Maps • Methods (Mapping Techniques) • Constructing A Map • Fill-In-The-Node Map • Fill-In-The-Line Map • Many Others • Data structure and scoring methods depend on the technique Shavelson & Ruiz-Primo: Windows Into The Mind

7. ConceptMap is a graph consist of consist of labeled lines nodes linked by denote a represent between a pair of concepts relation What Are Concept Maps?A Concept Map of a Concept Map! Shavelson & Ruiz-Primo: Windows Into The Mind

8. Concept Map Assessment Components • Task that invites a student to provide evidence bearing on his or her declarative knowledge structure in a domain • Format for the student’s response • Scoring system by which the student’s concept map can be evaluated accurately and consistently Shavelson & Ruiz-Primo: Windows Into The Mind

9. Are There Different Types of Concept Maps? Yes, more than we would like! Variations in the task, the response format, and the scoring system produce different concept mapping techniques. Shavelson & Ruiz-Primo: Windows Into The Mind

10. Concept Map Assessment: An Example Task The following map shows the relationship among concepts related to genetic continuity. The concepts are arranged hierarchically and linked each other. Please examine the map and supply, in the space provided, a word or two for labeling each link such that association between concepts is made clear. Response Format Scoring System Genetic continuity • One point is given for each correct relationship • One point is given for each level up until two levels beyond the last branching • One point is given for the first branching where two or more concepts are connected to the concept above. Three points are given for any subsequent branching • One point is given for each cross link Asexual reproduction Sexual reproduction Meiosis 1. through 2. producer 3. requires 4. called 5. fuse into Female gamete Male gamete Source: Tamir, P. (1995). Science Assessment. In M. Birenbaum & F. J. R. C. Dochy (Ed.). Alternatives in assessment of achievements, learning processes, and prior knowledge. Boston: Kluwer Academic Publishers.12 Shavelson & Ruiz-Primo: Windows Into The Mind

11. Another Example Task Construct a concept map that reflects what you know about Ions, Molecules, and Compounds. The map should have 10 concepts in it. Organize the terms in relation to one another in any way you want. Response Format Scoring System Focuses on the accuracy of the propositions: • Four points if the proposition is outstanding and complete • Three points if the proposition is correct and complete • Two points if the proposition is incomplete but correct • One point if the proposition is correct but does not show understanding between the two concepts. • Zero if the proposition Blank page for constructing the map Shavelson & Ruiz-Primo: Windows Into The Mind

12. On The Reliability & Validity of Concept Maps • Reliability: Can a student’s concept maps be consistently scored by different persons? • Equivalence of Mapping Techniques: Do all mapping techniques provide the same picture of a student’s declarative knowledge structure? • Task Demands: Do different mapping techniques impose different cognitive demands on a student? • Comparing Concept Maps With Traditional Assessments: Do concept maps measure something different than multiple-choice tests? Shavelson & Ruiz-Primo: Windows Into The Mind

13. Study 1:Tasks Mapping Technique 2 Mapping Technique 1 No-Concepts Provided Concepts Provided • Construct a concept map about Ions, Molecules, and Compounds. • Select another 7 concepts that you think are the most important in explaining the topic. • Organize the concepts • Construct a concept map about Ions, Molecules, and Compounds. • Examine the 10 concepts and construct the map. • Organize the concepts • Redraw the map so someone else can read it. LIST OF CONCEPTS acids anions ... Shavelson & Ruiz-Primo: Windows Into The Mind

14. Response Format Construct-a-Map: Concepts Provided Construct-a-Map: No Concepts Blank page to construct the map Concepts: Atoms, Compounds, Ions... Blank page to construct the map Shavelson & Ruiz-Primo: Windows Into The Mind

15. Scoring System Based on a criterion map, scoring focuses on: • Proposition Accuracy:Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent • Convergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map • Salience: Proportion of valid propositions to the total number of propositions in student’s map Shavelson & Ruiz-Primo: Windows Into The Mind

16. Proposition “Quality” Quality of Proposition Definition 4 points Excellent Outstanding proposition. Complete and correct. It shows a deep understanding of the relation between the two concepts. acids-compounds: < that gives off H+ when dissolved in water are Good Complete and correct proposition. It shows a good understanding of the relation between the two concepts. acids-compounds: > are examples of 3 points Incomplete but correct proposition. It shows partial understanding of the relation between the two concepts. acids-compounds: < form 2 points Poor Don’t Care 1 points Although valid, the proposition does not show understanding between the two concepts. acids-compounds: > is a different concept 0 points Incorrect proposition. acids-compounds: > made of Invalid Shavelson & Ruiz-Primo: Windows Into The Mind

17. Generalizability(Reliability) of Scores Percent of Total Variability and G Coefficients for Proposition Validity Total Scores No-Concepts, Sample A, and B Sample A & B • 71.64 78.67 • .15 0 • 0 0 • 0 .79 • 22.81 17.64 • .01 .18 • 5.37 2.69 • .89 .88 • .89 .88 • .78 • .78 Person (p) Rater (r) Sample (s) p x r p x s r x c prc,e     2 (nr=2,ns=3) (nr=2,ns=2) 2 (nr=1,ns=1) Shavelson & Ruiz-Primo: Windows Into The Mind

18. Discriminant Validity Correlation Between Multiple-Choice Test and Concept Map Scores Type of Score No-Concepts Sample A Sample B Proposition Validity Congruence Salience .64 .66 .61 .63 .55 .50 .58 a .45 Shavelson & Ruiz-Primo: Windows Into The Mind

19. Study 2:Tasks Construct-a-Map Fill-in-the-Nodes Fill-in-the-Linking Lines • Construct a map. • Use the 20 concepts provided to construct the map. • Organize the concepts anywayyouwant. • Check the map. • Redraw the map so someone else can read it. • Examine the skeleton map, the blank nodes and the terms provided. • Select the term that corresponds to each blank node. • Write the term inside the circle. • Check that all blank nodes are filled-in. • Examine the skeleton map, the blank linking lines and the linking words provided. • Select the linking words that correspond to each blank linking line. • Write the linking words on the blank linking line. • Check that all blank lines are filled-in. Shavelson & Ruiz-Primo: Windows Into The Mind

20. Response Format Construct-a-Map Fill-in-the-Linking Lines Fill-in-the-Nodes Elements, Atoms, and Compounds Elements, Atoms, and Compounds atoms atoms lose or gain electrons to form composed of composed of Blank page to construct the map Concepts: Elements, Atoms, Compounds... compounds ions compounds formed with cations and anions are formed from molecules sharing made up of opposite formed from molecules sharing made up of opposite ionic compounds molecular compounds that begins with H are with 3 or more elements are with 3 or more elements are example of N2O4 ternary ionic acids N2O4 Shavelson & Ruiz-Primo: Windows Into The Mind

21. Construct-A-Map Construct a concept map that reflects what you know about what a concept map is. Organize the terms in relation to one another in any way you want. Shavelson & Ruiz-Primo: Windows Into The Mind

22. Fill-In-The-Node Examine the map, the blank nodes and the terms provided in the list below. Select the term that corresponds to each node and write it down inside the circle. Use each term only once. Shavelson & Ruiz-Primo: Windows Into The Mind

23. Fill-In-The-Linking-Lines Examine the map, the blank lines and the linking words provided on the list below. Select the word(s) that corresponds to each line and write the number down in the line. Use each number only once. Shavelson & Ruiz-Primo: Windows Into The Mind

24. Scoring System Construct-a-Map Fill-in-the-Map • Proposition Accuracy:Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent • Convergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map • Salience: Proportion of valid propositions to the total number of propositions in student’s map • Correct/Incorrect:Student’s responses on the skeleton map were scored as correct or incorrect. Shavelson & Ruiz-Primo: Windows Into The Mind

25. Are Fill-In-The-Map Scores Sensitive To The Nodes/Linking Lines Selected To Be Filled-In? Means and Standard Deviations by Type of Map and Sample Mean S.D. Type of Skeleton Map n (Max = 12) Fill-in-the-nodes Sample 1 80 11.21 1.42 Sample 2 72 10.80 1.74 Fill-in-the-linking lines Sample 1 78 9.77 2.74 Sample 2 73 8.99 3.09 Shavelson & Ruiz-Primo: Windows Into The Mind

26. Are The Two Forms Of Fill-In-The-Map Equivalent? Means and Standard Deviations by Type of Map and Sequence Fill-in-the-linking lines Fill-in-the-nodes Sequence Mean S.D. Mean S.D. n 1 Nodes 1-Lines 1 43 11.09 1.52 9.72 2.84 2 Nodes 1-Lines 2 36 11.03 1.33 9.31 3.06 3 Nodes 2-Lines 1 35 10.63 1.81 9.83 2.65 4 Nodes 2-Lines 2 37 10.97 1.67 8.68 3.13 Total 152 11.02 1.59 9.36 2.93 Shavelson & Ruiz-Primo: Windows Into The Mind

27. Can The Two Mapping Techniques Be Considered Equivalent? Means and Standard Deviations across the Three Types of Assessments Type of Assessment Max n Mean S.D. Construct-A-Map Proposition Accuracy 152 135 53.91 22.17 Convergence 152 1 .50 .19 Salience 152 1 .73 .17 Fill-In Fill-in-the-nodes 152 12 11.02 1.59 Fill-in-the-lines 151 12 9.39 2.93 Multiple-Choice Test 150 30 24.05 3.74 Shavelson & Ruiz-Primo: Windows Into The Mind

28. Do The Mapping Techniques Provide Information Similar To That Provided By A Multiple-Choice Test? Structure Extent C-M FI-N FI-L MC Type of Assessment Construct-A-Map (C-M) (.99) Fill-in-the-nodes-NOD (FI-N) .47 (.71) Fill-in-the-lines-LIN (FI-L) .44 .40 (.85) Multiple-Choice Test (MC) .44 .37 .53 (.74) Shavelson & Ruiz-Primo: Windows Into The Mind

29. Other Empirical Evidence • Correlations tell you about rank ordering on different measures--that’s only part of the story • Evidence is needed for the cognitive claims that different methods measure somewhat different aspects of achievement: • Talk aloud • Focus group • Group work • Other? Shavelson & Ruiz-Primo: Windows Into The Mind

30. Talk Aloud with Concept Maps: Method Variation Correspondence Between Directedness and Inferred Activity Low Directed Proportion Scores High Directed Verbal Codes Shavelson & Ruiz-Primo: Windows Into The Mind

31. Tentative Concept Maps Conclusions • Students’ maps can be consistently scored by different persons even when complex judgments, such as proposition quality, are required • Different mapping methods provide different pictures of students’ declarative knowledge structure. The construct-a-map method provides opportunities to reveal students’ conceptual understanding • Different mapping techniques impose different cognitive demands on students. Highly structured mapping techniques allow students to respond by elimination or guessing, whereas constructed response do not. • Correlations between concept-map and multiple-choice scores are positive and moderate suggesting that these two types of assessment measure overlapping but somewhat different aspects of declarative knowledge Shavelson & Ruiz-Primo: Windows Into The Mind

32. Cognitive Maps • Methods • Word Association • Similarity Judgment • Card Sorting • Tree Building • Data structure: Proximities or distances • Scaling methods: Networks or hierarchical clusters Shavelson & Ruiz-Primo: Windows Into The Mind

33. Example: Similarity Judgments Rate the similarity of each pair of terms by circling one of the numbers provided or “DK -- Don’t Know 1. Central Tendency Mean (Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK 2. Hypothesis Description (Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK Shavelson & Ruiz-Primo: Windows Into The Mind

34. Concept x Concept Proximity Matrix For A Student Shavelson & Ruiz-Primo: Windows Into The Mind

35. Network Representation of Proximities Variance Variability Statistics Description Mode Sample Central Tendency Median Inference Population Mean t-test Hypothesis Regression ANOVA Correlation Shavelson & Ruiz-Primo: Windows Into The Mind

36. Small Pilot Study (N=28) Of Similarity Judgments And: Terms: Variability, Variance, Standard Deviation • Concept maps • Multiple choice Variability measures measures Std. Dev. Variance is the square root of 1. Which of the following refers to a procedure for making generalizations from a limited random sample of data? a. Statistical Inference b. Intuition c. Population Parameterization d. Standardization Source: Ed Wiley’s 2nd Year Project Shavelson & Ruiz-Primo: Windows Into The Mind

37. Sample Portion Of Concept Map refers to relationships within the Hypothesis is used to test population-related Sample is used to test hypotheses in addresses ideas about variables in population in the process of is used to test ideas about population in Linear Regression is a procedure in statistical uses sample to test ideas in the Population Inference is one special case of the general linear model, along with is a procedure in statistical ANOVA Shavelson & Ruiz-Primo: Windows Into The Mind

38. Correlation Among Methods Multiple Choice Concept Map Type of Assessment Pretest Concept Map .155 Similarity Judgment .554* .251 Posttest Concept Map .330 Similarity Judgment .434* .706* * p < .01 Shavelson & Ruiz-Primo: Windows Into The Mind

39. What If We Could? • Collect cognitive structure information indirectly by computer • Immediately show a student a representation of her knowledge structure • Permit her to modify the structure • And label the lines? Shavelson & Ruiz-Primo: Windows Into The Mind

40. Conclusions • Concept and cognitive maps appear to tap a different aspect of declarative knowledge than do multiple-choice tests • Concept and cognitive maps may provide similar representations of structure although this has not been tested directly • By assessing different aspects of knowledge, we may broaden what gets taught! Shavelson & Ruiz-Primo: Windows Into The Mind