Sean P. Goggins, Drexel University Krista Galyen , University of Missouri

1. Network Analysis of Trace Data for the Support of Group Work: Activity Patterns in a Completely Online Course Sean P. Goggins, Drexel University Krista Galyen, University of Missouri James L. Laffey, University of Missouri

2. Two Interwoven Problems With Completely Online Learning Groups • How do different activity types shape the social organization and practices of completely online CSCL Groups? • How bi-directional (as opposed to uni-directional) activity logs make group interactions and activity types more visible.

3. What Did We Find? • Bi-Directional Data Matters: If we just look at post data, we don’t see the richness of the network. Variance in Density, Centralization, Betweenness & other key social network measures do not discriminate until we add read data. • Activity Type: We do see connections between the activity types and the network statistics and structures • New Meanings for Old Statistics (Your Data Really Does Matter): For example, the difference between lurkers and influencers/connectors: High Betweenness Centrality has two semantics in online log analysis.

4. Outline • Related Work • Motivations • Socio-technical context & research methods • Analysis & Findings

5. Related Work Log Analysis Studies (sample) • Aviv, R., Erlich, Z., and Ravid, G. 2005. Response Neighborhoods in Online Learning Networks: A Quantiative Analysis. Educational Technology and Society. 8, 4, 90-99. • de Laat, M., Lally, V., Lipponen, L., and Simons, R.-J. 2007. Investigating Patterns of Interaction in Networked Learning and Computer-Supported Collaborative Learning: A Role for Social Network Analysis. Computer Supported Collaborative Learning. 2007, 87-103. • Reffay, C. and Chanier, T. 2003 How Social Network Analysis Can Help to Measure Cohesion in Collaborative Distance Learning. In Designing for Change in Networked Learning, Kluwer Academic Publishers. Basic Theory (sample) • Wasserman, S. and Faust, K. 1994 Social Network Analysis: Methods and Applications. Cambridge University Press • Katz, N., Lazer, D., Arrow, H., and Contractor, N. 2004. Network Theory and Small Groups. Small Group Research. 35, 307-332. • Koku, E. F. and Wellman, B. 2004. Scholarly Networks as Learning Communities: The Case of Technet. Designing for Virtual Communities in the Service of Learning. 338-376.

6. Gaps in Prior Studies • Descriptions of what “online” means is ambiguous in at least one important way: Proportion of work online, length of working relationship description, and/or description of socio-technical environment [we will be clear] • Use of uni-directional log data, and failure to reconcile log data with the experience in the user interface [we do] • Absence of examination of changes to networks over time [we do] • As a collection, the unit of analysis is highly variable [remains an issue] • May uniformly have validity issues [we address methodologically]

7. Network Analysis: Validity Issues • Historically Analyzed Using: • Self reported connections or publicly observed connections • In the physical world. Crowston, K., Howison J., & Wiggins A. (Submitted). Validity issues in the use of social network analysis for the study of online communities.

8. Motivations • Small groups who come together completely online are an emerging form of group interaction that is little studied. • In general, we think completely online graduate level courses provide an opportunity for the study of highly distributed, temporally bound collaboration by strangers. • We have developed tools that make the construction of context more visible for online learners • Our goal is to expand on that by making group behavior visible from multiple perspectives • Though most theories of groups and learning acknowledge the importance of passive behavior (listening & observing), the online corollary (reading [lurking]) is usually not incorporated in studies of online groups.

9. Research Questions • RQ1 - How do social interactions among completely online group members vary by activity type? • RQ2 - How and to what extent d0 the structure and patterns we see in qualitatively analyzed data also have visible representation in the sociograms? • RQ3 - How do bi-directional logs (as opposed to uni-directional logs) contribute to the understanding of social interactions and activity types seen in the analyses?

10. Socio-technical Context & Research Methods • Setting: Completely online graduate level software design course – students *never* meet face to face • Methods: • Network analysis {more detail shortly} • Content Analysis • Open & Axial Coding of Discussion Boards & Interview transcripts

11. System Description

12. System Description: Context Awareness Users know where the action is and who is involved in the action

13. Logging of CANS Transactions Reads & Posts: Raw Counts Bi-Directional Online SNA Data

14. The Two Problems Read is gray. Post is Blue. Bi-directional Data • How do different activity types shape the social organization and practices of completely online CSCL Groups? • How bi-directional (as opposed to uni-directional) activity logs make group interactions and activity types more visible.

15. Context Aware Logging Read Read course Person Module Post Group DB Person Post Page

16. Exploding Log Data • We “explode” the raw CANS data to reflect the real interactions. (how many discussion posts were on a page when it was viewed?) • In this sense, we are connecting the log data we analyze to the way information was experienced during interaction with our system. • Time weighted • If you viewed five posts, you have five connections, not just one • Recency matters (qualitative analysis suggests a four day “cliff” in the significance of an interaction in this data set) • Connections are reconstructed for all posts displayed on a page when a user reads the page (but weighted for time, as noted above)

17. Context Aware Logging course Read Read Person Read Read Module Read Read Post Read Read Read Group Read DB Read Read Read Read Person Post Page

18. Activity Description Tools & Tasks

19. Context Description: Activities & Activity Types

20. Analysis and Findings Analysis A note about our application of open coding, axial coding and other methods that emerge from grounded theory. We did not go into the field without particular theories in our minds or questions in our proposal. We were concerned about social structure, and the nature of relationships between participants. We use methods from grounded theory because we do not assume that instruments developed for use in the study of physically situated groups apply cleanly to the relationships we observe in completely online groups.

21. Network Density • The proportion of ties in a network compared the the maximum number of possible ties.

22. Analysis & Findings • RQ1 - How do social interactions among completely online group members vary by activity type? Network Density

23. Centralization A centralized network will have many of its links dispersed around a few nodes, while a decentralized network will see more equal link distribution around each node. In degree centralization captures this for read behavior Out degree centralization captures this for post behavior

24. Analysis & Findings • RQ1 - How do social interactions among completely online group members vary by activity type? Network Centralization

25. Betweenness • High betweenness people bridge clusters • In physically situated networks, betweenness is understood to be a sign of influence… high betweenness people are “connectors” in the physical world.

26. Analysis & Findings • RQ1 - How do social interactions among completely online group members vary by activity type? Highest betweenness members

27. Analysis & Findings • RQ2 - How and to what extent do the structure and patterns we see in qualitatively analyzed data also have visible representation in the sociograms? Peer to Peer Small Group Individual Clique analysis provides the sharpest contrast

28. Analysis & Findings • RQ3 - How do bi-directional logs (as opposed to uni-directional logs) contribute to the understanding of social interactions and activity types seen in the analyses? Analysis is relatively impoverished if we only include the post data that most systems use. We know that groups two and five were the most cohesive and performed the best (based on qualitative data analysis). Using core-periphery analysis, we see that these users were also most highly connected within the overall course network.

29. Analysis & Findings • RQ3 - How do bi-directional logs (as opposed to uni-directional logs) contribute to the understanding of social interactions and activity types seen in the analyses? Highest centralization members for the course overall

30. Analysis & Findings • RQ3 - How do bi-directional logs (as opposed to uni-directional logs) contribute to the understanding of social interactions and activity types seen in the analyses? Betweenness Means Something different online. Betweenness as Influence Betweenness as legitimate peripheral participation.

31. What Did We Find? • Bi-Directional Data Matters: If we just look at post data, we don’t see the richness of the network. Variance in Density, Centralization, Betweenness & other key social network measures do not discriminate until we add read data. • Activity Type: We do see connections between the activity types and the network statistics and structures • New Meanings for Old Statistics (Your Data Really Does Matter): For example, the difference between lurkers and influencers/connectors: High Betweenness Centrality has two semantics in online log analysis.

32. Next Steps • Implications for design • Groups are often not aware they are groups in completely online courses; few systems provide context data back to users. • When users see the data we do provide, they do not get a sense of the structure or presence of a group; they simply see a snapshot of a day. We can build better tools that make small group social context more visible • Implications for research • Applications of network analysis to trace data from electronic systems has limitations not often addressed by empirical studies. • The connection between user experience and log analysis is important when describing connections between users.

33. Questions? Thank You!