Problems, and little victories, in transcript analysis

1. Problems, and little victories, in transcript analysis Patrick J. Fahy Athabasca University, Centre for Distance Education CIDER Presentation 30 November 2007

2. Outline • Why TA? • Validity issues • Ethics principles • Implications for practice

3. Why transcript analysis? • A conference transcript is “a gold mine of information concerning the psycho-social dynamics” among the participants (Henri, 1992,p. 118). • CMC is related both to achievement and attitudes (Fulford & Zhang, 1993). • CMC is a way to establish instructor and social presence – facilitators of online community (Rourke, Anderson, Garrison, & Archer, 1999). • Community is “important to the success of” online learners (Conrad, 2005).

4. Why TA? (con’t) • CMC is common,but how to manage CMC, and how CMC-based interaction improves learning and online experience, is not well understood. • Community constructed - somehow - through language, in CMC(Vrasidas & McIsaac,1999). • “Further study is needed to describe the discourse devices used in CMC to establish community” (Lapadat, 2007, p. 62)

5. Validity - issues Lack of discriminant capability • Many postings in few categories (Gunawardena, Lowe, & Anderson, 1997; Kanuka & Anderson, 1998). • Postings could be coded into more than one category; category ambiguity (Zhu, 1996).

6. Validity – more issues Instruments too complex, too many codes • Gunawardena et al. (1997): 20 categories (5 “phases”) • Cookson & Chang, 1995: 16 codes • Rourke et al., 1999: 12 indicators • Higgins (1998): 20 • Zhu (1996): 8

7. Unreliability • inter-coder reliability is low • Sometimes, reliability data not reported

8. Questionable solutions • Collaborative (group) coding (engineered “coder drift”). • Mystical terminology • “units of meaning,” “discourse units,” “message units,” etc. • Coding of whole messages into 1 category • “One code fits all”

9. Considerations regarding validity • Principle of over-determination • Principle of commensurate complexity • Dangers of Type 1 errors

10. Principle of over-determination Social events usually have more causes than are strictly required for them to occur. • Contrary to Occam’s Razor, simplest explanations maynot be truest • “Seek simplicity, then distrust it.” (Whitehead) • “... human development is the result of many overlapping spheres of influence....” (Gibson, 2000) • “Unlike rare diseases such as Huntington’s, where a single genetic variation guarantees that a carrier will be afflicted, common diseases are triggered by a complex array of factors, including multiple genes each exerting a modest effect.” [Genes for several common diseases, (2007, September/October).

11. Principle of commensurate complexity • Social principles cannot usually be at once general, accurate, and simple. [Thorngate, W. (1976).]

12. A special problem Type 1 error: • rejection of a null hypothesis that is, in fact, true.

13. Ethics • In Canada, the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS, 2003), applies to all research, funded or not. • Developed and jointly issued by the Medical Research Council of Canada (MRC), the Natural Sciences and Engineering Research Council of Canada (NSERCC), and the Social Sciences and Humanities Research Council of Canada (SSHRCC).

14. Problems • Research ethics boards (REBs) and researchers disagree on what is “ethical.” • ... “an underlying current of distrust ... exists within the [Humanities] community vis-à-vis the TCSP” (Owen, Robert, Burgess, Golfman & Sykes, 2001, p. 4). • “Review boards pose threat to social scientists’ work.” (The Wall Street Journal, 2004)

15. Where is the real problem? • Ethics issues in clinical/biomedical research • Tuskegee syphilis study (1932 – 1972) • Polio vaccine studies on children (1950s) • Willowbrook hepatitis studies (1963 – 1966) • Drug trials (Olivieri vs. Apotex, 1998) • Others?

16. Key ethical principles:Getting minimal risk research approved • Ethics: from ethos = “custom.” • TCPS: REBs to apply ethical principles “in the context of the nature of the research and of the ethical norms and practices of the relevant research discipline” (TCPS, p. i.9). • Emphasize maximization of benefits, minimization of risks, potential harms.

17. Respect for human dignity • Persons are ends, not means. • Individual rights normally take precedence over the advancement of knowledge • Informed consent assures respect for individual rights... but collective rights are recognized, too.

18. Justice, inclusiveness • No group should bear an unfair burden in, or enjoy disproportionately the benefits of, research.

19. Understand informed consent (The “Gold” standard ) Conditions under which informed consent may be waived (TCPS, p. 2.1): • The research is minimal risk. • The waiver is unlikely to adversely affect the rights and welfare of the subjects. • The research could not practicably be carried out without it. • Subjects will be provided with additional information after participation, “whenever possible and appropriate.” • A therapeutic intervention is not involved [normally precluded by point 1].

20. Rationale • Majorities have rights: • Individuals may refuse to participate, and subjects who express concerns about a study may have the option of removing their data from the project; however: • This approach should be used only when the elimination of the subject’s data will not compromise the validity of the research design, and hence diminish the ethical value of participation by other subjects. (TCPS, p. 2.3; emphasis added)

21. So long as the methodology is sound, all contributions are anonymous, and identities are protected in any reports, it may well be a greater ethical violation, in relation to participation, to deny the rights of the willing majority than it would be to accede to the wishes of the reluctant minority by refusing to approve a research proposal. • “Nothing in this Policy should be interpreted to mean that research subjects have the right to veto a project, though they do, of course, have the right to refuse to cooperate with the researcher(s)” (TCPS, p. 1.2; emphasis added).

22. Other limitations to informed consent • Preserve the integrity of the research, by avoiding “colour[ing] the responses of the subjects and thus invalidat[ing] the research” (TCPS, pp. 2.2 – 2.3).

23. Individuals and groups not entitled to informed consent (Remember: this is the “gold standard” in the ethical treatment of human subjects) • Public bodies, corporations, political parties • Private organizations • “... may not interfere with or veto research about their own conduct, their participation in any research about their behavior is not required, and they need not be approached for their consent prior to research being conducted.” (TCPS, p. 2-2) • Public figures • (“Certain types of research … may legitimately have a negative effect on public figures ….” (TCPS, p. 1-6) • Naturalistic observations • Classroom, course evaluations • Quality assurance, performance reviews

24. Secondary use of data • “Secondary use of data refers to the use in research of data contained in records collected for a purpose other than the research itself. Common examples are patient or school records or biological specimens, originally produced for therapeutic or educational purposes, but now proposed for use in research. This issue becomes of concern only when data can be linked to individuals, and becomes critical when the possibility exists that individuals can be identified in the published reports.” (TCPS, p. 3.4; emphasis added)

25. The real gold standard of ethical research ANONYMITY.

26. Suggestions for practice (praxis) • Notice who speaks, who doesn’t. • Invite the reluctant. • Respect the desire for anonymity. • Notice hubs, spokes. • Assure hubs are positive; check on the isolated (people no one is talking or responding to). • Talk to everyone at least once • But don’t insist others do this. • Link, document. • Be the literature/social link.

27. Be present. • “80% of getting ahead is just showing up.” (Woody Allen) • Give credit for participation. • What gets measured gets done. • Set a standard. • What gets modeled gets done.

28. Citations • Cookson, Peter S. & Chang, Yu-bi. (1995). The Multidimensional Audioconferencing Classification System (MACS). American Journal of Distance Education, 9(3), 18-36. • Fulford, C. P. & Zhang, S. (1993). Perception of interaction: The critical predictor in distance education. The American Journal of Distance Education, 7(3), pp. 8 – 21. • Genes for several common diseases. (2007, September/October). Technology review, 110(5), p. 109. • Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global on-line debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), pp. 395-429. • Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 117-136). Berlin: Springer-Verlag. • Medical Research Council of Canada. (2005). Tri-council policy statement: Ethical conduct for research involving humans. Ottawa: Government of Canada. Available: http://www.pre.ethics.gc.ca/english/policystatement/policystatement.cfm

29. Oshinsky, D. M. (2005). Polio: An American story. Toronto: Oxford University Press. • Owen, M., Robert, L., Burgess, J., Golfman, N. & Sykes, S. (2001). Report to the Social Sciences and Humanities Research Council of Canada, Implementation of the Tri-Council Policy Statement on Ethics in Human Research (TCPS) (Humanities Project). Unpublished paper, Humanities and Social Sciences Federation of Canada. • Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), pp. 50-71. • Thorngate, W. (1976). “In general” vs.. “it depends”: Some comments on the Gergen-Schlenker debate. Personality and Social Psychology Bulletin 2, pp. 404 – 410. • Vrasidas, C. & McIsaac, M. S. (1999). Factors influencing interaction in an online course. American Journal of Distance Education, 13(3), pp. 22 – 36. • Zhu, E. (1996). Meaning negotiation, knowledge construction, and mentoring in a distance learning course. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology (18th, Indianapolis, IN). Available from ERIC documents: ED397849.

30. Pat Fahy Athabasca University, Centre for Distance Education patf@athabascau.ca 866-514-6234