An approach for measuring the convergence of group mental models

1. An approach for measuring the convergence of group mental models by Roy Clariana (rbc4@psu.edu) & Darryl Draper, INSYS PhD candidate (dcd120@psu.edu) Presented at the Annual "Collaborating Across Disciplines" PSGV Research Symposium, April 23, 2010

2. The Study Design Nonrandom Assignment (self selection) Participants are all Subaru Dealership ‘shop’ coordinators … Control Sub Group (n = 56) Community of Practice Sub Group (n = 65)

3. Community of Practice A group of individuals who share similar interests and through interaction and activities collectively develop new practices and knowledge. Wenger (1991)

4. Community of Practice Caveat 3 humorous examples (or perhaps not so humorous)

5. Community of Practice Caveat

6. Community of Practice Caveat

7. Cooperation is not collaboration • Collaboration, Mutual engagement of participants in a coordinated effort to complete a project or goal • Cooperation, Division of labor, each person is responsible for a portion of the project or goal Cooperation is more ‘efficient’

8. “...collaboration is the process of shared creation: two or more individuals with complementary skills interacting to create a shared understanding that none had previously possessed or could have come to on their own. Collaboration creates a shared meaning about a process, a product, or an event. In this sense, there is nothing routine about it. Something is there that wasn’t there before.” (p. 140) Michael Schrage (1990)

9. “In my teaching experience, cooperative learning represents the best means to approach mastery of foundational knowledge. Once students become reasonably conversant, they are ready for collaborative, ready to discuss and assess,....” Rockwood (1995)

10. So, putting it together… “The individual plays three roles in a collaborative environment: as independent agent, as cooperative agent, and as collaborator. Collaboration makes use of the group in order to increase information content and its filtering (many sets of eyes, hands and ears are better than one) and to construct a product. The ultimate goal of Collaboration, however, is not the product, but the learning of the individual student.  It is, in fact, essential that the individual be able to apply critical thinking to group ideas and conclusions.” E. Sky-McIlvai (2004)

11. Learning and workplace trends … • CSCL – Computer supported collaborative learning • CSCW – Computer supported cooperative work • CWE – collaborative working environment • CLW (?) – And Charles Findley presented the concept of collaborative learning-work (for teams) I believe that when our courses require Collaborative Learning, this engendersCollaborative Work on the job

12. Measuring team knowledge • Measure stuff while the team is in action (observations, video capture and post analyze communications, post analyze written communication, and so on…) • Measure extra stuff from the individuals who comprise the team (‘unnatural’ artifacts) • Measure some team created natural artifacts • Measure other stuff +++ +

13. Measuring associations as a proxy for knowledge structure • free word association • All pair-wise comparisons • Concept maps (note: not hierarchical) • Mindmaps • Causal diagrams • My KU-Mapper approach (sorting and list-wise rating) • 20 others interviews essays tests observations

14. Analyze and Represent… • After trying various approaches, I settled on Schanveldt’s Pathfinder networks as a way to analyze and represent the weighted associations that I was collecting using pair-wise tasks, sorting task, list-wise tasks, mindmaps, and now, essays • Pathfinder converts weighted association data into a least weighted path and generates a network diagram

15. For more info, see … • Clariana, R.B. (2010). Deriving group knowledge structure from semantic maps and from essays. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 7, pp. 117-130). New York, NY: Springer. link • Clariana, R.B. (2010). Multi-decision approaches for eliciting knowledge structure. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 4, pp. 41-59). New York, NY: Springer. link • A PFNET describes a vector of ‘1s’ and ‘0s’ and so that allows for lots of fun things: the similarity of PFNETS can be compared, PFNETS can be combined, multiple PFNETS can be represented as positions in an n-dimensional space

16. So putting it all together: collaboration, convergence, is a ‘coming together’ Individual’s PFNET Individual’s PFNET Individual’s PFNET N-dimensional semantic space

17. The Study Design Nonrandom Assignment (self selection) Participants are all Subaru Dealership ‘shop’ coordinators … Control Sub Group (n = 56) Community of Practice Sub Group (n = 65)

18. Scenario 1: Essay prompt Your Role. You are a Service Advisor for a dealership. Your responsibilities include the following: • Scheduling service appointments • Writing Repair Orders • Communication with Technicians • Communication with Customers Background. The dealership is located in a growing suburban area. You are consistently scheduling service appointments two weeks in the future. There are two other Service Advisors who work in the Service Department with you. Continued next slide 

19. Scenario 1: Essay prompt The Situation. One of the Service Advisors is out of the dealership due to a health issue. The Service Department is extremely busy with many customers waiting, and the phones are constantly ringing. A customer has been waiting for about 15 minutes. The vehicle was towed into the dealership. The repair will take approximately 3 hours to complete. The customer would like a loaner car for the day. There are no loaner vehicles available at the time. How would you respond to the customer’s personal and practical needs? Use the customer service skills presented in the course to write your story. Please incorporate your own experiences in your response.

20. Moodle Excel data file of Scenario 1 Essay 1 2 3 etc. (download the Moodle excel data and number each response)

21. Converting essays to network diagrams Scenario 1 essay prompt 121 essays ALA-Reader 121 prx arrays KNOT 121 PFNETs MS Word terms.txt terms.txt Mega spell check http://textalyser.net/ Frequency counts http://textalyser.net/ more geese…

22. (the company dogma as expressed by the instructor) The expert’s scenario 1 essay The Service Advisor must address personal and practical needs of the customer. The personal need is the most important in a situation like this. The Service Advisor must apologize by saying “that he/she” is sorry for the 15-minute wait before their interaction began. The customer’s car was towed in unexpectedly and his/her emotions are high, more than if they had scheduled an appointment for an oil change. Secondly, the Service Advisor should demonstrate to the customer sincere empathy about the situation by listening carefully, taking notes, and maintain eye contact and self-esteem. . The practical need is the business need and why the customer is at the dealership. Involving the customer in the decision-making process gives some control to the customer regarding the method of transportation other than the desired loaner car. The service advisor might say ”that he/she is very sorry that a loaner car is not available at this time, however, a representative from the dealership would be more than happy to drive you to your destination and pick you up when the customer is finished or the car is repaired. The repair will take about three hours to complete and our best technician will be repairing your car. If you prefer you can wait in the waiting room, or there is a nice restaurant next door if you are hungry.” Then explain to the customer that they are valued and thank the customer for being patient, helping through the process and their business. prx array data essay PFNET ALA-Reader KNOT

23. The expert’s scenario 1 essay The Service Advisor must address personal and practical needs of the customer. The personal need is the most important in a situation like this. The Service Advisor must apologize by saying “that he/she” is sorry for the 15-minute wait before their interaction began. The customer’s car was towed in unexpectedly and his/her emotions are high, more than if they had scheduled an appointment for an oil change. Secondly, the Service Advisor should demonstrate to the customer sincere empathy about the situation by listening carefully, taking notes, and maintain eye contact and self-esteem. . The practical need is the business need and why the customer is at the dealership. Involving the customer in the decision-making process gives some control to the customer regarding the method of transportation other than the desired loanercar. The service advisor might say ”that he/she is very sorry that a loanercar is not available at this time, however, a representative from the dealership would be more than happy to drive you to your destination and pick you up when the customer is finished or the car is repaired. The repair will take about three hours to complete and our best technician will be repairing your car. If you prefer you can wait in the waiting room, or there is a nice restaurant next door if you are hungry.” Then explain to the customer that they are valued and thank the customer for being patient, helping through the process and their business. prx array data essay PFNET ALA-Reader KNOT terms.txt

24. The expert’s scenario 1 essay This is what ALA-Reader sees service – needs – customer – need – situation – service – apologize – wait – customer – car – appointment – service – customer – situation – contact – need – need – customer – customer – gives – customer – loaner car – service – loaner car – available – time – customer – car – repaired – repair – take – complete – repairing – car – wait – waiting – explain – customer – thank – customer prx array data essay PFNET ALA-Reader KNOT terms.txt

25. The “service – needs” link service – needs – service – needs – customer – need – situation – service – apologize – wait – customer – car – appointment – service – customer – situation – contact – need – need – customer – customer – gives – customer – loaner car – service – loaner car – available – time – customer – car – repaired – repair – take – complete – repairing – car – wait – waiting – explain – customer – thank – customer prx array data essay PFNET ALA-Reader KNOT terms.txt

26. Scenario 1 expert essay as prx data expert essay response  prx array data essay PFNET ALA-Reader KNOT

27. Scenario 1 expert essay as a PFNET expert essay response  expert prx  prx array data essay PFNET ALA-Reader KNOT

28. Scenario 1 expert essay as a PFNET The high degree nodes are important for interpretation, so I show these above with thick lines

29. Comparing network diagrams Scoring individual essay PFNETs Pattern match to expert w/KNOT 121 PFNETs 121 scores Expert Essay PFNET Looking for Group Mental Model Convergence

30. Comparing network diagrams Scoring individual essay PFNETs Pattern match to expert w/KNOT 121 PFNETs 121 scores Expert Essay PFNET Scores = links in common with the expert essay. expert CoP 7 27 6 21

31. Comparing network diagrams Scoring individual essay PFNETs Pattern match to expert w/KNOT 121 PFNETs 121 scores Expert Essay PFNET Looking for Group Mental Model Convergence CoP even (avg. of 32 prx ) CoP odd (avg. of 33 prx ) 1 expert & 4 group PFNETs Average w/KNOT 121 prx arrays KNOT SP even (avg. of 28 prx ) Pattern match all pairs w/KNOT SP odd (avg. of 28 prx ) random assignment 10 scores

32. Looking for Group Mental Model Convergence How to average multitpleprx files KNOT software reads prx files in this format (1 + 0) / 2 = 0.5

33. Looking for Group Mental Model Convergence How to average multitpleprx files DATA: avg.prx similarities 29 items 1 decimals 0.1 min 1 max lower triangle: KNOT software averages the cell contents (1 + 0) / 2 = 0.5

34. Even and odd groupmental model convergence expert CoP odd 23 10 19 Observation: perhaps CoP are a little more like the expert, but a lot more like each other

35. Even and odd groupmental model convergence Pearson r of the raw prx data (435 element arrays) The 435 element raw prx data are similar to the PFNet data

36. Expert network diagram

37. Self Paced Odd ----------- Even CoP

38. Questions? • We have used this approach with • concept maps • pair-wise • list-wise, and clustering • text (essays) • e.g., we have compared students essay to their concept map; we have compared students concept maps to the lesson text that they read; we have compared students pair-wise responses to their list-wise and clustering responses