Conversations and Quality

1. Conversations and Quality Aiming for Educationally Valuable Talk in Online Discussions Sedef Uzuner Ruchi Mehta Educational Theory and Practice University at Albany State University of New York

2. Introduction “A worthwhile educational experience is embedded within a community of inquiry that is composed of teachers and students – the key participants in the educational process” (Garrison et al, 2000:88). It is within this community of inquiry that learning occurs when participants use dialogueto question, draw inferences, make connections and validate knowledge about matters of significance (Larreamendy-Joerns & Leinhardt, 2006).

3. Review of literature Previous studies that investigated talk in online discussion threads have focused on determining the quality of conversations based on critical thinkingconstructs. (See for example, Wickersham & Dooley, 2006; Jeong, 2003; Gilbert & Dabbagh, 2005; Hara, Bonk,& Angeli,2000; Garrison, Anderson, & Archer, 2001)

4. Our focus This study is intended to analyze and characterize student posts in online discussion threads based on the concept of educationally valuable talk (EVT) – a new concept for determining the educational value of dialogue taking place in online discussions.

5. Conceptual framework Educationally valuable talk (EVT) is an evolving construct, but the underlying basis for this notion remains firmly girded in Lev Vygotsky’s socio-cultural theory of learning : • knowledge is socially constructed; • knowledge building is created between/among individuals in their collaborative meaning-making through dialogue.

6. On dialogue and knowledge building… Are all conversations concerned with knowledge building? Sometimes linguistic transactions that take place in educational contexts are nothing more than mere interactions that are not sufficient to make construction or co-construction possible.

7. On dialogue and knowledge building… Are some interactions educationally more desirable than others? Quality is a polemical issue and judging any one kind of talk as a better form of communication than others is not the best practice. However, we argue that there are some talk types that are educationally more valuable, and therefore, more desirable than others.

8. Mercer’s “Exploratory talk” In an attempt to investigate the talk of primary school children working together at the computer, Mercer (1994) coined the term “exploratory talk”to refer to a style of interaction in which “hypothesis are proposed, objections are made and justified, and new relevant information is offered” (27). With his notion of “exploratory talk”, Mercer suggests that individuals learn better when their current view of knowledge is challenged, reformed, and elaborated through their interaction with others.

9. Educationally valuable talk (EVT) EVT refers to a particular interactional pattern in online discussion threads characterized as dialogic exchanges whereby participants collaboratively display constructive, and at times, critical engagement with the ideas or key concepts that make up the topic of an online discussion, and build knowledge through reasoning, articulation, and reflection. In other words… Learning takes place when individuals engage in exploratory transactions through language.

10. Educationally less valuable talk (ELVT) In opposition to EVT, the term educationally less valuable talk (ELVT) refers to talk that lacks substance in regards to critical and meaningful engagement with the formal content or the ideas that are put forward in the posts of others in an online discussion.

11. EVT indicators in online discussions

12. EVT indicators continued…

13. Some cautionary notes … • These proposed EVT indicators are not in hierarchical order. • Determining the educational value of a discourse is subjective. Talk that does not incorporate these indicators may have potential educational significance in a specific context. Likewise, talk that does not seem to be educationally valuable for some may have significance for others.

14. ELVT indicators

15. Some cautionary notes… • To achieve knowledge building, each post has to be a significant addition to the learning community’s pool of knowledge. Although not unimportant, experientialposts that include mere narratives or descriptions of personal experiences without reflection are not considered to be sufficient to meet this requirement. Therefore, such posts fall under the category of educationally less valuable talk • Posts that include judgmental and reproductionalindicators are also of lesser educational value because they are brief posts that lack substance in regards to critical and meaningful engagement with the formal content or the ideas that are put forward in the posts of others in an online discussion.

16. Some cautionary notes continued… • Although posts that include affective indicators add social presence to the discussions, they do not have any teaching value; therefore, they do not allow participants to build knowledge. • As the name itself suggests, miscellaneous posts are interactions that are not concerned with knowledge building; therefore, they do not have any potential educational value.

17. Summary of our conceptual framework There is no talk type that is unimportant; however, there are some that are educationally more valuable and desirable than others. With the production of EVT, social interactions in online discussion threads will “progress beyond information sharing to knowledge construction” (Anderson et al, 2001: 9) and transform into purposeful and constructive dialogues which move toward the improved understandings of all parties involved.

18. Research Questions 1) What are the nature and characteristics of talk generated by learners in online discussions? 2) What are the indicators of educationally valuable talk in online discussion threads and how often do students engage in such conversations? 3) What factors seem to facilitate or hinder the production of educationally valuable talk in online discussions?

19. Methodology: Context of the study • A master’s level course offered through The State University of New York’s online instructional program - SLN (SUNY Learning Network) - provided the context of this study. • The course was co-taught by a professor and a teaching assistant (the first presenter) • It consisted of 7 modules that dealt with issues related to language, literacy and technology. • Participation in this course was asynchronous and the instructors placed a great deal of value in the discussion component of the course.

20. Methodology: Participants 19 graduate students participated in the course: • 4 male & 15 female • 4 were international students from Korea & China • 13 had experiences taking online courses before • 2 were PhD students

21. Methodology: Sources of Data The study draws on data collected from a variety of sources: 1. Students’ online posts (N = 706) from 3 separate modules (Only the messages posted in the whole-classdiscussions area were analyzed, because in other learning activities (such as group/pair discussions), students were given the option to conduct their discussions on IM, the phone, or other means) 2. Direct observations Certain features of the course including the instructors’ pedagogical approach to facilitating discussions, learning activities, and other course happenings were used to obtain information on the situational context in which students’ talk occurred. 3. Informal interviews with the course instructors 4.Students’ retrospective analysis

22. Data analysis • Content analysis was chosen as the main methodology. The content analysis of data focused on manifestation of EVT and ELVT in students’ posts • The base unit of analysis was a message (post). • When a message contained more than one indicator, they were each counted as separate indicators. And, one instance of a code category in a message was agreed to be sufficient to assign that code

23. Methodology: Procedures • One of the investigators is the co-instructor of the course under study. To reduce potential bias, she conducted the study with another investigator, a graduate assistant who had no connection with this course or participants in it. • No information is revealed to the course participants regarding the study and its purposes. • The posts of the following people are excluded from the data analysis: a) The instructors b) The 2 PhD students who knew about the study and its purposes

24. The coding scheme • Discussions were printed after taking out individual student names and any other personal information. • Using EVT and ELVT as a framework, the first presenter made a list of coding categories which was used as the preliminary coding criteria. • Both investigators independently coded the messages from the first module based on this preliminary criteria and then they met to reach a consensus on the coding scheme.

25. The coding protocol For each module, the coding protocol was conducted in two stages: Stage 1: The investigators coded the messages independently by using the coding scheme/instrument that they have agreed on using. Stage 2: The investigators met as a second time to discuss their findings to establish the extent of consensus on the use of the coding instrument (Cohen’s kappa is also used)

26. Inter-rater Reliability (Cohen’s kappa) Cohen's kappa (as described by Anderson et al, 2001) is a chance-corrected measure of inter-rater reliability that assumes two raters, n cases, and m mutually exclusive and exhaustive nominal categories. The formula for calculating kappa is: k = (Fo - Fc) / (N - Fc) where: N = the total number of judgments made by each coderFo = the number of judgments on which the coders agreeFc = the number of judgments for which there is no agreement.

27. Inter-rater Reliability Inter-rater reliability scores were as follows: After Module 1: 0.73 After Module 3: 0.87 By convention, a Kappa> .70 is acceptable inter-rater reliability, but this depends highly on the researcher’s purpose. Another thumb of rule is that K= 0.40 to 0.59 is moderate inter-rater reliability; 0.60 to 0.79 is substantial, and 0.80 is outstanding. (Landis & Kock, 1977)

28. Results 1) What are the nature and characteristics of talk generated by learners in online discussions?

29. Total Talk

30. Amount of talk: Indicator Share

31. Results 2) What are the indicators of educationally valuable talk in online discussion threads and how often do students engage in such conversations?

32. Proportion of Talk: EVT and ELVT

33. Some numbers.. Percentage of talk in Modules: A comparison Module1 Module 2 Module 3 Total EVT 80.5 81.4 83 81.4 ELVT 19.5 18.5 17 18.5

34. What factors…..??? What factors seemed to facilitate the production of educationally valuable talk in online discussions? 1. Class norms generated by students On the first week of class, the students were asked to generate ideas to define the class rules for this course. These norms the students came up with presented themselves in all modules as a reminder of class expectations

35. A synthesis of class norms: • Discussions should be well-organized, concise, relevant, and meaningful. • Posts should contain interesting ideas and questions that will stimulate the discussion and move it forward. • In addition to saying “I agree”, participants should accurately state why and give some facts or personal accounts to back it up. • The atmosphere of the discussion sessions should be supportive and trustworthy with posts conforming to proper etiquette. Posts that do not agree with the general consensus should be welcomed. • Posts should be made in a timely manner. Participants should not wait until the very end of an assignment to post their responses. • Postings that stem from participants’ diverse backgrounds are welcomed, but they should not rely heavily on personal opinions and experiences. • Ideas and experiences should be tied to the course readings whenever possible. • When a participant attaches an outside reading to the discussion, he/she should give a brief overview of it.

36. Factors continued… 2. Grice’s Maxims: Starting from Module # 1, students were asked to rate their posts according to Grice's Cooperative Principle of Conversation (Grice, 1975.) (1 = my posting reflects this maxim 100%; 4 = I could do much better when it comes to this maxim) • Quantity: make your contribution as informative as is required, but not more, or less, than is required. • Quality: do not say that which you believe to be false or for which you lack evidence. • Relation: be relevant. • Manner:avoid ambiguity and obscurity, be clear, brief and orderly.

37. Students’ perceptions… What are students’ perceptions of the quality of talk that occurred in whole-class discussions? Excerpts from students’ retrospective analysis: “The discussions are enlightening and I often walk away from my computer thinking about the ideas being introduced. […]My posts need to be much better developed and clearer.” “I am usually very careful about writing my posts, with regards to both the content and the manner, so I would have to say that I am quite satisfied with the quality of my posts. For the most part, I am quite happy about others’ posts as well”

38. Students’ perceptions continued… “In general, the posts in this course have been better than what I've seen in some of the other courses. There is less of "fluff 'n stuff" or essentially vacuous replies posted for requirement's sake only”

39. Student perceptions How did the students feel about using Grice’s maxims and establishing class norms on their own? “The norms and Grice's maxims compelled me to expand on other postings, reflect on my own experience, connect to course content, explore possibilities, or challenge ideas. I think that class norms and criteria for postings are essential components to quality educational discussions. I think that our class has done a great job creating quality postings. As I was looking back on our past conversations I found many examples of EVT. Personally, I learned a great deal from my classmates due to the consistent quality of postings.” “Overall, we kept this online discussion very interactive, almost no discussion went without several responses and most responses were relevant and well thought-out because there were guidelines that kept us from answering just yes or no. Grice’s maxims are good guidelines for the online course discussion”

40. What was the instructors’ perspective? The instructors found the participation in the course to be well above average and were pleased with the frequent and quality contributions students made to the discussions.

41. Teaching Implications 1) Have students establish class norms on online discussions 2) Have them use Grice’s maxims to evaluate their own posts as well as those of their peers

42. Teaching Implications: Suggested activity: In Module 6, the students were introduced to these two constructs: educationally valuable talk (EVT) & educationally less valuable talk (ELVT). Once they read and discussed the conceptual underpinnings for each talk type, they were asked to select and analyze a conversation/discussion thread of their choice that exemplify interaction which leads to learning and broadening understanding (EVT).

43. Teaching Implications/ Suggested Activity To conduct this assignment, the students were given the following options for their selection: 1. Select sample exchanges from any past module of this course 2. Select sample exchanges from another online course they were taking. 3. Select sample exchanges from Internet sites where some form of tutoring or instruction is the objective (the subject matter is irrelevant so you might find conversations about fixing a vacuum cleaner or analyzing a Shakespearian sonnet etc) 4. Select sample exchanges from another source they may have (check with the instructors first)

44. Student reactions to this assignment: “I found this assignment [conversation analysis] to be rewarding and worthwhile. It may be a good starter to an on-line course. What do you think?”

45. Next steps: Suggestions for future research This research can be further expanded into a treatment-control group study by comparing discussions in modules with treatment (Class-norms and Grice’s maxims) with the ones without it.

46. References • Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001). Assessing Teaching Presence in a Computer Conferencing Context. Journal of Asynchronous Learning Networks, 5 (2), 2-17. • Ebel, Robert L. (1951). Estimation of the reliability of ratings. Psychometrika 16: 407-424. • Garrison, R., Anderson, T., & Archer, W. (2000). Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education. The Internet and Higher Education, 2 (2-3), 87-105. • Gilbert, P. K. & Dabbagh, N. (2005). How to structure online discussions for meaningful discourse: a case study. British Journal of Educational Technology, 36 (1), 5-18. • Hara, N., Bonk, C., Angel, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28, 115-152. • Jeong, C. A. (2003). The Sequential Analysis of Group Interaction and Critical Thinking in Online. American Journal of Distance Education, 17(1), 25-43.

47. References continued… • Kumpulainen, K. (1996). The nature of peer interaction in the social context created by the use of word processors. Learning and Instruction, 6(3), 243-261. • Landis, J. R., Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33:159-174. This article sets cut-offs for Cohen's Kappa. • Larreamendy-Joerns, J. & Leinhardt, G. (2006). Going the Distance With Online Education. Review of Educational Research,76 (4), 567-605. • Mercer, N. (1994). The quality of talk in children’s joint activity at the computer. Journal of Computer Assisted Learning, 10, 24-32. • Vygotsky, L.S. (1962). Thought and language. Cambridge, MA: MIT Press. (Original work published 1934) • Wickersham, L.E., & Dooley, K.E. (2006). A content analysis of critical thinking skills as an indicator of quality of online discussion in virtual learning communities. Quarterly Review of Distance Education, 7(2),185-193.

48. Questions? Comments? Sedef: usedef@yahoo.com Ruchi: mruchi@gmail.com Educational Theory and Practice University at Albany State University of New York