Using ALA-Score software to score essays and concept maps

1. Using ALA-Score software to score essays and concept maps Wednesday June 22, 2005 Dr. Roy Clariana Penn State University email: RClariana@psu.edu Web: www.personal.psu.edu/rbc4 "First we build the tools, then they build us!" -- Marshall McLuhan

2. goals • Your take away: • Some understanding of how/why it works • Some ideas that you could implement on Monday morning in your classroom (or in you research) • Hands-on using ALA-Score software to score student essays • Hands-on using CMAP tools concept map software • Quick interest survey (teacher, researcher, grade level, subject, writing?, concept maps?)

3. Roy • H.S. math and science teacher (4 years) • E.S. technology teacher (5 years) • Ed. Consultant in CO, WY, UT (5 years) • College professor at Penn State (8 years) www.personal.psu.edu/rbc4 • The grand kids live in Denver and we visit our cabin in summers here

4. Important note • Concept maps and essays are proven powerful generative instructional approaches! • But the main focus of this presentation is assessment rather than instruction, both ongoing in class (formative) and at the end of units of instruction (summative)

5. The “structure” of knowledge • Most tests measure declarative knowledge (e.g., knowing who, what when, where, why and how, etc.) • But essays and concept maps can measure the organization or structure of knowledge essays interviews tests observations

6. Eliciting structural knowledge • Vygotsky (in Luria, 1979); Miller (1969) card-sorting approaches • Deese’s (1965) ideas on the structure of association in language and thought • Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysis • Recent neural network representations (e.g., Elman, 1995)

7. Deese, free recall data • moth? – ___________ • yellow? – ___________ • bug? – ___________ Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

8. Deese, free recall data (p.56) Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

9. Simple visual representation from Multi-Dimensional Scaling of this Deese data Novices will not have a developed knowledge structure

10. concept map example Bahr (2004) using concept maps to teach English to German students

11. Side trip • Wordnet: http://wordnet.princeton.edu/ http://wordnet.princeton.edu/cgi-bin/webwn • What is the Visual Thesaurus? – The Visual Thesaurus offers a unique visual display of the English language. Looking up a word creates an interactive visual map with your word in the center of the display, connected to related words and meanings. • For example, type “bird” in at: http://www.visualthesaurus.com/trialover.jsp

12. Tools to support concept mapping • Yellow stickies!! Pencil and paper may be best for your classroom • Software – PowerPoint is pretty good • Inspiration is good but expensive • CMAP tool is free, but your tech person will have to agree to support it • At least 22 other tools are available, some free some not (see next slide)

13. Additional concept map automatic scoring approaches • CMap tools (IHMC) that we will use today • C-TOOLS – Luckie (PI), University of Michigan NSF grant available: http://ctools.msu.edu/ctools/index.html • TPL-KATS – University of Central Florida (e.g., Hoeft, Jentsch, Harper, Evans, Bowers, & Salas, 1990). TPL-KATS: concept map: a computerized knowledge assessment tool. Computers in Human Behavior, 19 (6), 653-657. • SEMNET – http://www.semanticresearch.com/about/ • CMAT – Arneson & Lagowski, University of Texas, http://chemed.cm.utexas.edu • Plus 22 other non-scoring map tools

14. Some classroom uses of concept mapping • Usually involve individuals working alone, and involve in some way reading or writing text • Some collaborative strategies have been used • Lets look at a few to give you classroom ideas…

15. Using a student concept map to “capture” a text (i.e., text summary, note taking) concept map notes Textbook Text text text text text text text text text text text Textbook Text text text text text text text text text text text Textbook Text text text text text text text text text text text memo text text text text As Homework? Summary Text text text text text text text text text text text student

16. Using a concept map in place of an outline in the writing process concept map notes essay Text text text text text text text text text text text memo text text text text Examples? student

17. Using a student concept map to capture and organize research on a topic text Text text text text tex Text text text text textt concept map notes text Text text text text tex Text text text text textt memo text text www text text Examples? video Report Text text text text text text text text text text text student video

18. Using a team concept map to capture and organize research on a topic text Text text text text tex Text text text text textt concept map notes text Text text text text tex Text text text text textt memo text text www text text Examples? video Report Text text text text text text text text text text text team video

19. Example of dyad collaboration Note the attentional effects of the artifact Verbal discussion (taped) Observations: On task Abstract talk 3-propositions/min Question Answer Criticize Conflict Elaboration Co-construction concept map artefact Analyze the discussion Blah blah blah blah Blah blah The incredible value of talk! Blah blah blah blah Blah blah Hannah Inferred: Active use of prior knowledge Acknowledged problems Look for meaningful relations Negotiation Problem: Sometimes unscientific notions are ingrained Yergin Shared objects play an important role in negotiation and co-construction van Boxtel, van der Linden, Roelofs, & Erkens (2002)

20. summary • Maps  into text • Text  into maps • There is an intimate relationship between the two and both can provide measures of structural knowledge • Have you considered concept maps

21. ALA-Reader Text CMAP … an electrical signal starts the heartbeat, by causing the atrium to contract. The blood then flows through the pulmonary valve into the pulmonary artery and then into the lungs. Once inside the lungs, the blood gives up the carbon dioxide (cleansed) and receives oxygen. This oxygenated blood … atrium contract P valve P artery lungs cleansed oxygenated Link array

22. Clariana & Koul • Participants – A group of 24 practicing teachers enrolled in CI 400 • Lesson – while researching the topic “the structure and function of the heart” online, students completed concept maps using Inspiration software and later wrote an essay on this topic from their maps. Clariana, R.B., & Koul, R. (2004). A computer-based approach for translating text into concept map-like representations. In A.J.Canas, J.D.Novak, and F.M.Gonzales, Eds., Concept maps: theory, methodology, technology, vol. 2, in the Proceedings of the First International Conference on Concept Mapping, Pamplona, Spain, Sep 14-17, pp.131-134. http://cmc.ihmc.us/papers/cmc2004-045.pdf

23. Posttests Essays • Multiple-raters using holistic rubric • Computer-derived LSA Essay scores (http://www.personal.psu.edu/rbc4/frame.htm) Concept Maps • Multiple-raters using Lomask’s rubric • ALA-Mapper PFNet link and distance agreement with an expert • ALA-Reader PFNet link scores (from 1 to 5) (so far, only looked at essay scores) 

24. ALA-Rater PFNet scores • The scores for each text and rater-pair are shown ordered from best to worst. • ALA-Reader scores were moderately related to the combined text score, Pearson r = 0.69, and ranked 5th overall. ALA-Reader

25. other tools to score Essays • ETS – PEG (Project Essay Grade), e-rater, Criterion and other products… http://www.ets.org/research/erater.html • Walter Kintsch (and Landau) at CU-Boulder – Latent semantic analysis, many uses, i.e., score online training for the Army - http://lsa.colorado.edu/ • Vantage Learning essay scoring products - http://www.vantagelearning.com/ ALA-Reader: http://www.personal.psu.edu/rbc4/score.htm

26. other tools to score Essays • http://Knowledge-Technologies.com • http://www.ets.org/research/erater.html • http://ericae.net/betsy/ • http://torrseal.mit.edu/effedtech/ (the MIT physics homework tutor) • http://knowledge-technologies.com/hrw.html (Preparation for Standardized Writing Tests) • http://knowledge-technologies.com/KeysDemo/Keys2.1D.html (Prentice Hall – Using the IEA to improve textbook learning) • http://knowledge-technologies.com/tradoc.html (U.S. Army TRADOC – Automated Essay Scoring for Officer Leadership training)

27. Comments and Questions • About foundation of the idea and its use • Any ideas on how you have or will use maps? • Next: demo ALA-Reader

28. Demo ALA-Reader • Download ALA-Reader.exe from www.personal.psu.edu/rbc4 • Create terms file (can include 2 synonyms) • Create 2 expert baseline reference texts called expert1.txt and expert2.txt (i.e., Instructor, best student) • Use it • Files created • Summary file called report.txt • Multiple *.prx files (PRX folder) • CMAP files

29. DEMO of CMAP Tools • Lets walk through it together, follow me step by step

30. Example of an online collaboration concept map artefact concept map session lasted 80 minutes. 3 x 12 online groups, communicate by chat, 745 messages were exchanged (avg. of 62 per group). creates Online chat H: WE should … J: Did you see… Y: Yeah, but … Etc. Etc. Only the lead could alter the concept map Jari Hannah (lead) The ‘other 2 members used chat to “advise” Researchers Analyzed the chat text And the concept map Yergin p.22, Chiu, Huang, & Chang (2000)

31. Demonstration – Cmap tools synschronous collaboration (see the Project handout)

32. Brainstorm, then make the map • Open IHMC Cmap tools • Fill in personal information on first use (I’ll tell you what to type in here) • Click Other Places • Open brainstorm file • Click collaborate icon if necessary • Type in your first name • Collaborate

33. Demo 1 IHMC Public Cmaps conv v2 on Jan 22 2004

34. Demo 1 Oulu EDTECH Public

35. Final Questions • Time to play with the software

36. My research interests prototypes • Mind map assessment – automatic scoring software tool called ALA-Mapperhttp://www.personal.psu.edu/rbc4/ala.htm • Essay assessment – automatic scoring software tool called ALA-Readerhttp://www.personal.psu.edu/rbc4/score.htm • for Latent Semantic Analysis (LSA) see: http://www.personal.psu.edu/rbc4/frame.htm

37. #1st Poindexter and Clariana • Participants – 23 undergraduate students in intro EdPsyc course (Penn State Erie) • Food rewards for participation • Setup – complete a demographic survey and how to make a concept map lesson • Text based lesson interventions – instructional text on the “heart” with either proposition specific or relational lesson approach Poindexter, M. T., & Clariana, R. B. (in press). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), in press. link to doc file

38. Treatments • Relational condition, participants were required to “unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content • Proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided).

39. Posttests • Concept map (use 26 terms provided) • Link-based common scores • Distance-based common scores • Multiple-choice tests (Dwyer, 1976) • Identification (20) • Terminology (20) • Comprehension (20)

40. Map-link Map-dist Means and sd

41. Analysis • MANOVA (relational, proposition-specific, and control) and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc. • COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance. • Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table).

42. Correlations Map-link Map-link Map-distance All sig. at p<.05 Compare to Taricani & Clariana next 

43. Taricani and Clariana – Replication of Poindexter and Clariana TermComp Link data 0.78 0.54 Distance data 0.48 0.61 Taricani, E. M. & Clariana, R. B. (in press). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), in press.

44. Compare these two . . . Taricani & Clariana TermComp Link data 0.78 0.54 Distance data 0.48 0.61 Poindexter & Clariana TermComp Link data 0.77 0.53 Distance data0.69 0.71

45. # 2nd Clariana, Koul, & Salehi • Participants – A group of 24 practicing teachers enrolled in CI 400 • Lesson intervention – while researching online, completed concept maps in pairs (newsprint & yellow stickies) to describe the structure and function of the heart and then individually wrote essays on this topic from their maps. Clariana, R. B., Koul, R., & Salehi, R. (in press). The criterion related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33 (3), in press.

46. Posttests Essays • Multiple-raters using holistic rubric • Computer-derived LSA Essay scores (http://www.personal.psu.edu/rbc4/frame.htm) Concept Maps • Multiple-raters using Lomask’s rubric • ALA-Mapper PFNet link and distance agreement with an expert

47. Correlation matrix Human Computer Map Essay LSA Link Map 1 Essay 0.49 1 LSA 0.31 0.73 1 Link data 0.36 0.76 0.83 1 Distance data 0.60 0.77 0.71 0.82 1 p < .05 shown in boldface type. Many investigators have noted the close relationship between maps and essays.