Research Design and Tools for Internet Research

1. Research Design and Tools for Internet Research Claire Hewson University of Bolton ESRC e-society Programme and Sage Handbook of Online Research Methods Colloquium, 28-29 March, 2007

2. Aims • Summarise some key issues and principles involved in designing Internet-mediated research (IMR) studies • Consider some of the approaches possible (tools, procedures), and reasons for choosing between these alternatives • Highlight areas which need further exploration, evaluation and development in relation to IMR design principles

3. Research Design in IMR • Research design - the process of making choices at various levels, or stages, within the research (including developing a research question, specifying a theoretical orientation, selecting data collection and analysis procedures) • Research design in IMR - focus primarily on the data-gathering stage

4. Why Use IMR? • Advantages • cost / time efficiency (no data input costs, or multiple copies of materials, etc.) • easy access to potentially vast, geographically diverse participant pool (IUP) [enhanced reach, statistical power, access to small specialist populations] • unique features of online environment can lead to various potential benefits (e.g. reduced social desirability effects, increased candour, balanced power relationships, etc.)

5. Why Use IMR? • Disadvantages • biased nature of IUP(?) [inherent bias in Internet-accessed samples; generalisability issues] • reduced levels of control (over procedures / stimuli, participation context, participant behaviour) • features of online interaction could lead to various disadvantages (e.g. ambiguities in communication due to lack of extra-linguistic cues)

6. Tools, Procedures, Design Considerations in Quantitative IMR • Design choices essentially linked to the criteria of validity, reliability and generalisability • Surveys, experiments, observation, and document analysis, have been implemented to date

7. Internet Surveys • Tools: • www forms, email, ftp; lower or higher tech solutions (e.g. javascript, server-side scripting); audio / video capabilities • Issues influencing design choices: • levels of expertise / equipment available • levels of control required (over presentation / response format) • sophistication of procedures / functionalities required (ID-checking, etc.) • data security (validity / ethical concerns)

8. Internet Surveys • Overall, web-based approaches offer far greater scope and are likely to be preferable for most types of survey-based research, issues relating to levels of technical support / ease of implementation aside

9. Internet Experiments • Tools: • essentially same range of tools and approaches as for survey-based research, though web-based approaches likely to be required in many cases (precise timings, control over presentation, etc.) • Issues influencing design choices: • essentially, the same issues as with survey-based approaches emerge (ease of implementation, levels of control, functionalities required, data security)

10. Internet Experiments • Overall, web-based approaches are almost always to be preferred (and will often be necessary) in experimental IMR, given the levels of control and sophistication of functionalities that will often be required. Though in some research contexts alternative approaches may be possible (e.g. emailing experimental text-based materials to participants, e.g. Hewson, 1994)

11. Summary of IMR Design in Quantitative Research • A trade-off emerges between simple, low tech implementations which may suffer reliability and validity issues, and higher tech solutions which allow greater levels of control (etc.), but place greater demands on the researcher (and possibly participant, perhaps restricting access and thus affecting generalisability). Clearly specific research aims and goals, as well as practical considerations, will direct the design choices made.

12. Tools, Procedures, Design Considerations in Qualitative IMR • Design choices essentially linked to criteria of trustworthiness (dependability, etc.) • Interviews, observation, and document analysis, have been implemented to date

13. Internet Interviews • Tools: • asynchronous (email, mailing lists, discussion groups) and synchronous (IM, online chat) approaches; audio / video capabilities • Issues influencing design choices: • nature of interaction / data desired (rich, detailed, reflective; factual, quick to obtain; candid, sensitive, personal, etc. [relates to: timescale; levels of anonymity; availability of extra-linguistic information]) • demands placed on participant and researcher (skill / experience levels, etc.) • reach, access and diversity (e.g. issue of different time zones, accessibilty by e.g. disabled participants)

14. Internet Observation / Document Analysis • Tools: • essentially same range as for interviews, though often archives of online interactions will be most useful, which tend to involve asynchronous communications (real-time observation is also feasible, and observation beyond text-based interactions, e.g. in MU*s) • Issues influencing design choices: • specificity of material required (e.g. rumour transmission, Bordia, 1996) • resources / expertise available (e.g. chat software) • disclosed / undisclosed (relates to research aims, and ethics [e.g. public / private domain]) • naturalistic / solicited (e.g. Hessler et al, 2003)

15. Summary of IMR Design in Qualitative Research • Trade-offs will tend to emerge, e.g. asynchronous [depth, reflexivity, empowering participants, accessibility / reach] versus synchronous [enhanced flow, extra-linguistic information, speedier]. Greater anonymity may enhance candour, but inhibit rapport. Audio / video approaches currently suffer reliability issues. Cross-cultural approaches would seem to demand asynchronous methods. Naturalistic (archives) versus solicited approaches. Ethical considerations (disclosure, consent, anonymity / confidentiality).

16. Sampling Issues • Tools: • email, discussion group / mailing list postings, www adverts, dedicated IMR study pages, sampling offline • Issues: • inherent biases in IUP • probability sampling problematic (of IUP, and of sub-populations, e.g. dormant email addresses, etc.) • sampling procedures may lead to biases, but difficulty in measuring sampling frame / nature of biases • how to maximise response (follow-ups; incentives; issue salience; group readership / volume of postings) • ethics (netiquette, e.g. permission of moderators, topic relevance; email requests?)

17. IMR - Some Proposed Good Design Principles • Use procedures which can restrict variations on parameters which must crucially remain constant to ensure validity / reliability, but which maximise accessibility • Gather as much information as feasible to help detect unwanted variations / validity threats • Provide clear, explicit instructions, beyond what may normally be necessary for offline approaches • Pilot extensively (e.g. in different browsers)

18. IMR - Some Proposed Good Design Principles • Aim for procedures where sampling frame can be estimated, and sample bias assessed, when generalisability is important • Carefully consider ethical issues, especially when sensitive / personal information is being obtained • Consider features of synchronous and asynchronous interactions, and their effects, and effects of greater or lower levels of anonymity, and assess the relevant trade-offs in order to select procedures that best meet research aims / goals

19. Issues for Further Research in IMR Design • Validation studies which support IMR procedures (e.g. adaption of psychometric tests, experimental procedures) • Nature of IUP and relationship between sampling procedures and sample characteristics • Effects of online interactional medium • Issues surrounding public / private domain online • Reliability of technologies (e.g. audio / video) • Further piloting and exploration of advantages / disadvantages of novel approaches, e.g. mixed methods studies