Alternative measures of knowledge structure: a s measures of text structure and of reading comprehension May 14, 2012

1. Alternative measures of knowledge structure: as measures of text structure and of reading comprehension May 14, 2012 BSI Nijmegen, Nederland Roy Clariana RClariana@psu.edu Clariana, R.B. (2010). Multi-decision approaches for eliciting knowledge structure. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 4, pp. 41-59). New York, NY: Springer. link

2. Overview • Introduction • I am an instructional designer and a connectionist, so my language may be a little different, also slow me down if my accent is difficult • My intent today is to describe my research on several approaches for measuring Knowledge Structure (KS) and along the way, describe tools, and maybe show extra ways of thinking about text, knowledge, comprehension, and learning

3. KS: Encompassing theoretical positions • Cognitive structures (de Jong & Ferguson-Hessler, 1986; Fenker, 1975; Korz & Schulz, 2010; Naveh-Benjamin, McKeachie, Lin, & Tucker, 1986; Shavelson, 1972) • Conceptual networks (Goldsmith et al., 1991) • Conceptual representations (Geeslin & Shavelson, 1975; Novick & Hmelo, 1994); (McKeithen, Reitman, Rueter, & Hirtle, 1981) • Conceptual structures (Geeslin & Shavelson, 1975; Novick & Hmelo, 1994) • Knowledge organization and knowledge structures (McKeithenet al., 1981) • Semantic structures (Gentner, 1983; Riddoch & Humphreys, 1999).

4. KS: Encompassing theoretical positions • Spatial knowledge (de Jong & Ferguson-Hessler, 1996; Dunbar & Joffe, 1997; Jee, Gentner, Forbus, Sageman, & Uttal, 2009; Korz & Schulz, 2010; Schuldes, Boland, Roth, Strube, Krömker, & Frank, 2011) • Categorical knowledge (Candidi, Vicario, Abreu, & Aglioti, 2010; Matsuka, Yamauchi, Hanson, & Hanson, 2005; Stone & Valentine, 2007; Wang, Rong, & Yu, 2008) • Conceptual knowledge (de Jong & Ferguson-Hessler, 1996; Edwards, 1993; Gallese & Lakoff, 2005; Hallett, Nunes, & Bryant, 2010; Rittle-Johnson & Star, 2009)

5. KS: My sandbox model Our symbolic connectionist view: • Knowledge structure (or structural knowledge) refers to how information elements are organized, in people and in artifacts • A departure from most theories, we propose that knowledge structure is pre-propositional, but that KS is the precursor of meaningful expression and the underpinning of thought • Said differently, knowledge structure is the mental lexicon that consists of weighted associations (that can be represented as vectors) between knowledge elements

6. KS is worth measuring • Measures of content knowledge structure have been empirically and theoretically related to memory, classroom learning, insight, category judgment, rhyme, novice-to-expert transition (Nash, Bravaco, & Simonson, 2006) and reading comprehension (Britton & Gulgoz, 1991; Guthrie, Wigfield,Barbosa, Perencevich, Taboada, Davis, Scafiddi, & Tonks, 2004; Ozgungor & Guthrie, 2004), and • And findings for combining individual knowledge structures to form group mental models (Cureeu, P.L., Schalk, R., & Schruijer, S., 2010; DeChurch& Mesmer-Magnus, 2010; Johnson & O’Connor, 2008; Mohammed, Ferzandi, & Hamilton, 2010; Pirnay-Dummer, Ifenthaler, & Spector, 2010).

7. Applied to reading comprehension, KS as a measure of the situation model Ferstl & Kintsch (1999) • Textbase (the text’s semantic content and structure, van Dijk & Kintsch, 1983) • Situation model (the integration of the ‘episodic’ text memory with prior domain knowledge, van Dijk & Kintsch, 1983); also called mental model of the text, the text model, the discourse model Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.), The construction of mental representations during reading. Mahwah, NJ: Lawrence Earlbaum.

8. Visually Text base Situation model (pre list recall) Updated situation model (post list recall) Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.), The construction of mental representations during reading. Mahwah, NJ: Lawrence Earlbaum.

9. A KS measure of the situation model • Ferstl& Kintsch (1999) used pre-and-post-reading list-cued partially-free recall to elicit KS of the birthday story (which obtains asymmetric matrices) • Participants – 42 undergraduate students (CU Boulder) • Pre-reading cued-association KS task: Students were presented by computer a 60 word list of birthday-related terms to view one at a time (randomized), and then were given the list on paper with 3 blanks beside each list term and were asked to write in the 3 terms from the list that come to mind • Reading: Students then read the 600-word long birthday story • Post-reading cued-association KS task: i.e., same as pre-task, fill in the list Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

10. Results • Established that the KS cued association paradigm was appropriate for assessing background knowledge and text memory • This KS approach facilitated interpretation, depicting how the text ‘added to’ the post reading situation model (see their figure 10.4, p.260); provided a different or other way to think about reading comprehension (p.268) • Test-retest reliability may be a problem for this KS approach Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.), The construction of mental representations during reading. Mahwah, NJ: Lawrence Earlbaum.

11. Another KS measure of the text base (or situation model?) • Clariana& Koul(2008), we asked students to draw concept maps (KS) of a text • Participants – 16 graduate students in a science instructional methods course (Penn State GV) • First, students discussed concept maps in class • Then working in dyads (8 pairs), students were given a 255 word passage on the heart and circulatory system and were asked to create a concept map of it • KS data sources • 8 dyad concept maps of the text • 1 expert concept map of the text • A Pathfinder network (PFNet) map of the text automatically formed by ALA-Reader software Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

12. Data • 26 terms identified across all of the maps and text • (Text  concept map), dyads’ concept map link lines entered into a 26 x 26 half matrix • Matrix analyzed using Pathfinder Knot Link Array a b c d e f g a left atrium - b lungs 0 - c oxygenate 0 1 - d pulmonary artery 0 1 0 - e pulmonary vein 1 1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle 0 0 0 1 0 0 - (n2-n)/2 pair-wise comparisons Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

13. Data as percent overlap • Percent overlap was calculated as links in common divided by the average total links 2 54 e.g., Dyad PFNet e.g., Expert PFNet 4 % overlap = 4/ ((6+8)/2) % overlap = 4/ 7 % overlap = 57% Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

14. Data as percent overlap An aspect of measurement reliability and validity low good ALA-Reader  In the epigraph to Educational Psychology: A Cognitive View, Ausubel (1968) says, “The most important single factor influencing learning is what the learner already knows.” Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

15. The strong influence ofprior domain knowledge Figure 3. The relationship between the number of propositions in the dyad concept maps and the average percent agreement with the 255-word text passage (* shows dyads with a science major). Only those with prior domain knowledge could adequately ‘capture’ the text Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

16. ALA-Reader papers ALA-Reader converts text  KS Clariana, R.B., Wallace, P.E., & Godshalk, V.M. (2009). Deriving and measuring group knowledge structure from essays: The effects of anaphoric reference. Educational Technology Research and Development, 57, 725-737. Clariana, R.B., & Wallace, P. E. (2007). A computer-based approach for deriving and measuring individual and team knowledge structure from essay questions. Journal of Educational Computing Research, 37 (3), 209-225. Koul, R., Clariana, R.B., & Salehi, R. (2005). Comparing several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32 (3), 261-273. Clariana, R.B. (2010). Deriving group knowledge structure from semantic maps and from essays. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 7, pp. 117-130). New York, NY: Springer. • Also see HIMAT/DEEP software and Hamlet software

17. KS for influencing learning • e.g., Trumpower et al. (2010) used knowledge structure of computer programming represented as network graphs to pinpoint knowledge gaps • KS elicited as pair-wise comparisons and data-reduced to networks using Pathfinder KNOT • Learners’ networks then compared to an expert referent network Trumpower, D.L., Sharara, H., & Goldsmith, T.E. (2010). Specificity of Structural Assessment of Knowledge. Journal of Technology, Learning, and Assessment, 8(5). Retrieved from http://www.jtla.org.

18. KS for influencing learning • The problems were intended to be complex enough so that the solution depended on integration of several interrelated concepts (relational) • The presence of subsets of linksin participants’ PFnets differentially predicted performance on two types of problems, thereby providing evidence of the specificity of knowledge structure Trumpower, D.L., Sharara, H., & Goldsmith, T.E. (2010). Specificity of Structural Assessment of Knowledge. Journal of Technology, Learning, and Assessment, 8(5). Retrieved from http://www.jtla.org.

19. Protein structure as an analogy of knowledge structure in reading comprehension Christian Anfinsen received the Nobel Prize in Chemistry in 1972: • Linear sequence of amino acids  enzyme structure  enzyme function Is like: • Linear sequence of words in a text  knowledge structure  retrieval function

20. AA Linear sequence  enzyme structure  function APRKFFVGGNWKMNGKRKSLGELIHTLDGAKLSADTEVVCGAPSIYLDFARQKLDAKIGVAAQNCYKVPKGAFTGEISPAMIKDIGAAWVILGHSERRHVFGESDELIGQKVAHALAEGLGVIACIGEKLDEREAGITEKVVFQETKAIADNVKDWSKVVLAYEPVWAIGTGKTATPQQAQEVHEKLRGWLKTHVSDAVAVQSRIIYGGSVTGGNCKELASQHDVDGFLVGGASLKPEFVDIINAKH Triose Phosphate Isomerase: http://www.cs.wustl.edu/~taoju/research/shapematch-final.pdf

21. Read linear sequence of words in text Figure 1, p.136 Hyona, J., & Lorch, R.F. (2004). Effects of topic headings on text processing: evidence from adult readers’ eye fixation patterns. Learning and Instruction, 14, 131–152.

22. Knowledge structure Retrieval function A  B (propositional knowledge): Where do pandas live? In the wild A  B,C,D (relational knowledge): What do we know about pandas today? Pandas are heading towards extinction in the wild due to climate change Retrieval structure linear

23. Read KS  Retrieval function Retrieval function A  B (propositional knowledge): Where do pandas live? In the wild A  B,C,D (relational knowledge): What do we know about pandas today? Pandas are heading towards extinction in the wild due to climate change Retrieval structure Relational

24. Summary of the introduction • KS cuts across theories, we support connectionist views • KS is worth measuring, it correlates with many kinds of performance • KS can be measured in different ways • KS has been used to visually represent the reading comprehension situation model • KS has been used to visually represent the text structure • Specific KS structure leads to specific cognitive performance • Enzyme Analogy: linear chain  structure  function

25. Measuring knowledge structure My foundation and trajectory for measuring KS: • Vygotsky (in Luria, 1979); Miller (1969) card-sorting approaches • Deese’s (1965) ideas on the structure of association in language and thought • Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysis • Recent neural network representations (e.g., Elman, 1995) Jonassen, Beissner, and Yacci (1993) 

26. Dave Jonassen’s summary of KS measures… Trumpower, Sharara, & Goldsmith, 2010 similarity ratings Knowledge elicitation Ferstl& Kintsch, 1999 free recall concept maps written text Clariana & Koul, 2008 Knowledge comparison Knowledge representation Elicit responses  represent responses  compare response Jonassen, Beissner, & Yacci (1993), page 22

27. Dave Jonassen’s summary … To show different KR let’s do an example … word associations semantic proximity similarity ratings card sort relatedness coefficients Knowledge elicitation ordered recall quantitative graph comparisons scaling solutions graph building free recall C of PFNets concept maps written text qualitative graph comparisons Knowledge comparison additive trees Knowledge representation expert/ novice hierarchical clustering Trees Dimensional Networks MDS – multidimensional scaling ordered trees link weighted principal components cluster analysis minimum spanning trees Pathfinder nets Jonassen, Beissner, & Yacci (1993), page 22

28. Knowledge Representation (KR) • Multidimensional scaling (MDS) - Family of distance and scalar-product (factor) models. Re-scales a set of dis/similarity data into distances and produces the low-dimensional configuration that generated them (e.g., see: http://www.tonycoxon.com/EssexSummerSchool/MDS-whynot.pdf) • Pathfinder Knowledge Network Organizing Tool (KNOT) algorithms take estimates of the proximities between pairs of items as input and define a network representation of the items. The network (a PFNET) consists of the items as nodes and a set of links (which may be either directed or undirected for symmetrical or non-symmetrical proximity estimates) connecting pairs of the nodes. (See: http://interlinkinc.net/KNOT.html)

29. Pathfinder Network (PFNet) analysis • Pathfinder seeks the least weighted path to connect all terms, shoots for n-1 links if possible • Pathfinder is a mathematical approach for representing and comparing networks, see: http://interlinkinc.net/index.html • Pathfinder data reduction is based on the least weighted path between nodes (terms), so for example, Deese’s 171 data points become 18 data points. Only the salient or important data is retained. • Pathfinder PFNet uses, for example: • Library reference analysis • Use google to search to see many more examples of how Pathfinder can be used Note that Ferstl & Kintsch (1999) used Pathfinder

30. Deese (1965), free recall data (p.56) 100 participants are shown a list of related words, one at a time, and asked to free recall a related term Full array (n * n): 19 x 19 = 361 Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171 Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

31. Deese (1965), free recall data (p.56) Full array (n * n): 19 x 19 = 361 Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171 Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

32. Using MDS in SPSS • Start SPSS and open this Deese data file • Under Analyze, select Scale, then select Multidimensional Scaling (ALSCAL)… 1. Move Variable from left to right 2. Create distances from data 3. Model 4. Options How to - next page

33. Select all of these

34. MDS of the Deese data

35. Side issue, the MDS obtains alternate visual representations (e.g., enantiomorphism) Eindhoven Amsterdam Utrecht Nijmegen The Hague The Hague Nijmegen Utrecht Amsterdam Both are “correct solutions”. WARNING!! Eindhoven Like geographic data, for example, MDS may be oriented in different ways (describe Ellen Taricani’s 2002 dissertation, handing out teacher maps post-reading is a bad idea)

36. How good is the MDS representation for displaying the relationship raw data? • Many dimensions (in this case 19) reduced to 2 dimensions • Check the “stress” value to estimate how strained the results are MDS is an algorithmic, power, approach rather than based on a distribution model, so no assumptions about data structure are required…

37. sky summer blue spring sunshine garden yellow color flower nature butterfly cocoon moth wing bees bird fly insect bug PFNet of Deese data

38. MDS and PFNet of the exact same data from Deese SPSS MDS (i.e., global structure, relational, fuzzy, gist) Pathfinder KNOT PFNet (i.e., local structure, verbatim, proposition specific)

39. MDS and PFNet of the exact same data from Deese Blue lines reproduce the PFNet links SPSS MDS (i.e., global structure, relational, fuzzy, gist) Pathfinder KNOT PFNet (i.e., local structure, verbatim, proposition specific)

40. MDS and PFNet data reduction • MDS uses all of the raw data to reduce the dimensions in the representation; if the stress is not too large, global clustering is likely to be good but local clustering less so, and the MDS distances between terms within a tight cluster of terms are more likely to misrepresent the relatedness raw data. • Pathfinder uses only the strongest relationship data (typically 80% of the raw data is discarded). Pathfinder analysis provides “a fuller representation of the salient semantic structures than minimal spanning trees, but also a more accurate representation of local structures than multidimensional scaling techniques.”(Chen, 1999, p. 408)

41. Dave Jonassen’s summary … Sabine Klois used … word associations semantic proximity similarity ratings distance data card sort relatedness coefficients Knowledge elicitation ordered recall quantitative graph comparisons scaling solutions graph building free recall C of PFNets concept maps written text qualitative graph comparisons Knowledge comparison additive trees Knowledge representation expert/ novice hierarchical clustering Trees Dimensional Networks MDS – multidimensional scaling ordered trees link weighted principal components cluster analysis minimum spanning trees Pathfinder nets Jonassen, Beissner, & Yacci (1993), page 22

42. Poindexter and Clariana • Participants – undergraduate students in an intro Educational Psychology course (Penn State Erie) • Setup – complete a demographic survey and how to make a concept map lesson • Text based lesson interventions – instructional text on the “human heart” with either proposition specific or relational lesson approach • KS measured as ‘distances’ between terms in a concept map (a form of card sorting) and also concept map link data, but analyzed with Pathfinder KNOT Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

43. Treatments • Relational condition, participants were required to “unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content • Proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided). Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

44. DK and KS Posttests • DK - Declarative Knowledge (Dwyer, 1976) • Identification drawing test (20) • Terminology multiple-choice items (20), declarative knowledge, e.g., the lesson text states A  B, the posttest asks A  ?(B, x, y, z) (explicitly stated) • Comprehension multiple-choice items (20), inference required, e.g., given A  B and B  C in the lesson text, posttest asks A  ?(C, x, y, z) (implicit, not stated) • KS - Knowledge structure • Concept map link-based common scores • Concept map distance-based common scores Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

45. Note that declarative knowledge multiple-choice posttest items are sensitive to the linear order of the lesson text If the lesson text is A  B, paraphrasing the stem (A’) and/or transposing stem and response (B  A) to create posttest questions influences performance. When MC posttest is: • Identical to lesson (A  B): 77% • Transposed from lesson (B  A): 71% • Paraphrased from lesson (A’  B): 69% • Both T & P from lesson (B  A’): 67% posttest Bormuth, J. R., Manning, J., Carr, J., & Pearson, D. (1970). Children’s comprehension of between and within sentence syntactic structure. Journal of Educational Psychology, 61, 349–357. Clariana, R.B. & Koul, R. (2006). The effects of different forms of feedback on fuzzy and verbatim memory of science principles. British Journal of Educational Psychology, 76 (2), 259-270.

46. Recording link and distance data in a concept map (n2-n)/2 pair-wise comparisons Link Array left atrium a b c d e f g a left atrium - b lungs 0 - right ventricle to the c oxygenate 0 1 - d pulmonary artery 0 1 0 - pulmonary vein moves through e pulmonary vein 1 1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle 0 0 0 1 0 0 - pulmonary artery passes into Distance Array to the a b c d e f g a left atrium - lungs b lungs 120 - c oxygenate 150 36 - deoxygenated oxygenated d pulmonary artery 108 84 120 - e pulmonary vein 73 102 114 138 - f deoxgenate 156 42 54 84 144 - Student’s concept map g right ventricle 66 102 138 42 114 120 -

47. Distance raw data reduction by Pathfinder KNOT (21 distance data points reduced to 6 link data points) left atrium right ventricle pulmonary vein pulmonary artery Pathfinder Network Distance Array a b c d e f g a b c d e f g a left atrium - a left atrium - lungs b lungs 0 - b lungs 120 - c oxygenate 0 1 - c oxygenate 150 36 - d pulmonary artery 0 1 0 - deoxygenated oxygenated d pulmonary artery 108 84 120 - e pulmonary vein 1 0 0 - 0 e pulmonary vein 73 102 114 138 - f deoxgenate 0 1 0 0 0 - f deoxgenate 156 42 54 84 144 - g right ventricle 1 0 0 1 0 0 - Pathfinder network (based on distances) g right ventricle 66 102 138 42 114 120 -

48. Example of link and distance PFNets for the same concept map left atrium left atrium right ventricle right ventricle to the pulmonary vein pulmonary vein moves through pulmonary artery pulmonary artery passes into to the lungs lungs deoxygenated oxygenated deoxygenated oxygenated Student’s concept map (i.e., link data) Pathfinder network (from distance data)

49. Means and sd Map-link Map-dist Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

50. Analysis • MANOVA (relational, proposition-specific, and control) and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc. • COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance. • Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table). Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.