Roy B. Clariana Pennsylvania State University, USA RClariana@psu.edu

Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference Roy B. Clariana Pennsylvania State University, USA RClariana@psu.edu Patricia E. Wallace The College of New Jersey, USA PWallace@tcnj.edu Veronica M. Godshalk University of South Carolina, USA Godshalk@gwm.sc.edu Available at: http://www.personal.psu.edu/rbc4/CELDA.ppt

How do pronouns influence computer-based text analysis? • Participants in an undergraduate business course (N = 49, 4 missing data, final N = 45) completed an essay as part of the course final examination and investigators manually edited every occurrence of pronouns in these essays to their antecedents. • The original unedited essays (with pronouns) and the edited essays (no pronouns) were processed by ALA-Reader software using linear aggregate and sentence aggregate methods. These data were then analyzed using a Pathfinder network (PFNET) approach.

ALA-Reader text analysis ALA-Reader – free software, converts text to a network representation (open with CMAP tools from IHMC) and also outputs relationship data in the prx format that can be analyzed by KNOT Pathfinder network software. ( http://www.personal.psu.edu/rbc4)

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more. Example sentence from a student’s essay Key words are underlined Pronouns are shown in red ALA-Reader has two approaches, either linear aggregate or sentence aggregate

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more. job empowered job humanist satisfaction employee productivity productivity Here is the sentence aggregate run shown in black. Historical basis…an excel spreadsheet to do sentence aggregates

humanist job satisfaction productivity employee empowered ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more. Here is the linear aggregate run shown in black, so far so good, but what happens on the 2nd occurrence of ‘job’

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then they produced more. humanist satisfaction job . productivity employee . empowered

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then they produced more. satisfaction humanist job The force-directed graph begins to fold back on itself productivity empowered . employee

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more. The force-directed graph begins to fold back on itself

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more. Unedited essay – 7 links (pronouns present but invisible) Edited essay – 8 links (pronouns converted to referents)

ALA-Reader text analysis When pronouns are changed to their referent, the term Humanist (right figure) takes on central importance in the network (has the most links). Unedited essay – 7 links (pronouns present but invisible) Edited essay – 8 links (pronouns converted to referents)

½ array = prx file 29 terms used here ALA-Reader data output

ALA-Reader text analysis Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more. job empowered job humanist satisfaction employee productivity productivity Here is the sentence aggregate run shown in black. Sparse sentences under specified and dense sentences over specified, pronouns always under specify representations in the sentence aggregate approach.

ALA-Reader Sentence and linear aggregate output Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more. Linear aggregate prx output Sentence aggregate prx output

The essay writing promptand 29 key terms “Describe and contrast in an essay of 300 words or less the following management theories: Classical/Scientific Management, Humanistic/Human Resources, Contingency, and Total Quality Management. Please use the terms below in your essay: administrative principles, benchmarking, bureaucratic organizations, contingency, continuous improvement, customers, customer focus, efficiency, employee, empowerment, feelings, Hawthorne studies, human relations, humanistic, leadership, management (i.e., bosses), Management by Objectives, motivate, needs, organization (i.e., corporation), plan, product, quality, relationship, scientific management (classical), service, situation (or environment), TQM, and work (or job, task).”

Descriptive data regarding the essays • The 45 student essays averaged 301 words per essay (range from 170 to 479 words, standard deviation of 70.1). • The fifteen most common words account for 31% of all of the text and include in order the (6.2% of all words), and (3.3%), of (3.3%), to (3.1%), management (2.4%), a (1.9%), is (1.8%), in (1.6%), that (1.4%), employees (1.4%), this (1.1%), on (1.1%), are (0.9%), as (0.9%), and their (0.9%). • The three most common pronouns are their (ranked 15th. 0.9% of all words), it (18th, 0.8%), and they (19th, 0.8%). Students use of ‘their’ and ‘they’ in their essays indicates that they were writing from the perspective of either a manager or of an employee and not always as an independent observer.

prx output for student essays b07 and b10 (29 key terms) KNOT software reads prx files in this format

Example: prx average of student essays b07 and b10 It is possible (even easy) to use KNOT software to average together any number of individual student prx (essay) files in order to represent a team or group composite network representation; and then compare group-to-group and individual-to-group, etc. We compared top (n=15) and bottom (n=15) students.

Pathfinder KNOT • KNOT analyzes prx data • Based on graph theory, KNOT uses an algorithm to determine a least-weighted path that links all of the terms. The resulting PFNET (force-directed graph) is based on a data reduction approach that is purported to represent the most salient relationships in the raw proximity data.

Linear aggregate PFNET group average representation of the top 15 student essays 29 common links Unedited essay – 31 links (pronouns are not used) Edited essay – 34 links (pronouns converted to referents)

Unedited essay Edited essay It is fascinating to us to compare the PFNET visual depictions of the averaged essays. For example, the terms ‘management’, ‘employee’, ‘organization’, ‘customer’, and ‘product’ are well connected and so are central terms in both representations. Three of the four super-ordinate management theory categories in the essay writing prompt, ‘scientific management’, ‘contingency’, and ‘TQM’, were all associated with ‘management’ while the fourth category, ‘humanistic’, was associated with both ‘management’ and ‘employee’. Emotion-related terms such as ‘feelings’, ‘needs’, ‘relationship’, and ‘motivation’ were all associated with ‘employee’ in both PFNET representations as were the action words ‘benchmarking’, ‘work’, and ‘product’.

Comparing unedited and edited group representations Pearson correlations of the averaged proximity raw data (above the diagonal) and PFNETsimilarity data (below the diagonal in red) for the top group (n = 15) and the bottom group (n = 15) for both unedited essays and edited essays.

Individual essay scores • To calculate students’ essay scores, we used KNOT to compare the students PFNET to an expert PFNET. Student’s score consisted of the number of links in common with the expert. • A “universal” essay score for each student was established for comparison purposes using the SPSS factor score combination of the three human essay scores plus the ALA-Reader linear aggregate unedited essay score

Individual essay scores Correlations with ALA-Reader individual essay scores are shown in red. The length-adjusted ALA-Reader score correlation with the universal score (r = .84) is almost as good as that of the human raters.

Summary/Conclusions • The ALA-Reader sentence aggregate approach was not discussed much in this presentation, it didn’t perform well, but it is described in the manuscript • For the ALA-Reader linear aggregate approach, there was little difference between the PFNETs obtained for edited and unedited essays. Apparently the linear approach is relatively robust to pronoun interference and is generally superior to the sentence aggregate method for the narrow purposes of average group knowledge representation and for scoring individual essays.

Roy B. Clariana Pennsylvania State University, USA RClariana@psu.edu