Experimental Research Methods in Language Learning

1. Experimental Research Methods in Language Learning Chapter 8 A Hybrid Approach for Experimental Research

2. Leading Questions • What do you think is a hybrid approach? • Do you think it is adequate to rely on only one type of data (e.g., either quantitative or qualitative data)? Why do you think so? • Do you think it is easier to analyze qualitative data than quantitative data?

3. Hybrid Approach • A hybrid approach to experimental research arises from the influence of the mixed methods approach for combining statistical methods with qualitative ones. • A pragmatic approach of research methodology that experimental researchers avoid restricting themselves to using only a specific traditional method.

4. Hybrid Approach • A hybrid approach aims to combine the strengths of both quantitative and qualitative methods to gain a greater understanding of the influences of independent variables (e.g., instructions and environmental factors) on dependent variables (e.g., language learning and behaviors).

5. Three common Hybrid designs • To use one method to complement another • To use one research approach as the starting point for another • To use a quantitative method to answer a particular question, and a qualitative method to address others

6. Qualitative Data in Experimental Research • Qualitative data is information that can be described in words, rather than numbers. • Qualitative research generally uses data elicitation techniques, rather than instruments (e.g., think-aloud protocols, stimulated recalls, individual or group interviews, diaries, and spoken or written language). • Qualitative data helps researchers further explore or describe a phenomenon or process, and/or explain or exemplify an issue or a case.

7. Qualitative Research Techniques • Think-aloud protocols • Stimulated-recall and retrospective interviews • Individual and group interviews • Qualitative observations

8. Think-aloud Protocols • An introspective technique which allows researchers to have access to participants’ online cognitive processing or thinking, particularly higher-level thinking (see Ericsson & Simon 1993; Sasaki 2014). • Is useful for examining individual differences in cognitive processing. • Typically, researchers can analyze think-aloud data quantitatively (e.g. that data gained from participants answering ‘what’ questions) or qualitatively (e.g. that data gained from participants answering ‘why’ and ‘how’ questions).

9. Stimulated Recalls and Retrospective Interviews • Stimulated recalls and retrospective interviews are retrospective techniques that allow researchers to examine learners’ cognitive processes, thoughts, or feelings during task completion (see Gass & Mackey 2000). • Retrospection is a post-event verbal report. • Useful when researchers do not want to or should not interrupt learners’ naturally occurring processes during task completion

10. Individual and Group Interviews • Individual interviews are one-on-one interactions between an interviewer and interviewee • Group interviews (as with focus group) can be conducted with more interviewees at the same time (e.g., a group of 5). • Quantitative interviews, Qualitative interviews, Structured interviews, Semi-structured interviews, Open-ended interviews

11. Qualitative Observations • Allows researchers to observe, record, and describe what they see in a setting without having to tally frequencies of behaviors or check predetermined lists. • Focus on forming a holistic picture of an issue and aim to obtain a complete description of behaviors, interactions, cultural norms, values, attitudes, or social practice in a specific natural setting • Complete observers, complete participants, participants as observers, observers as participants

12. Quantitative Analysis of Qualitative Data • There is no single right way to analyze qualitative data. • Qualitative data such as that obtained through think-aloud protocols, stimulated recall and interviews can take numerical values through systematic data coding, and therefore can be used as quantitative data.

13. Units of Analysis • Such units include (Foster, Tonkyn and Wigglesworth, 2000) : • Semantic units (e.g., proposition, C-unit, and idea unit) • Intonational units (e.g., tone unit, idea unit, and utterance) • Syntactic units (e.g., sentence, idea unit, T-unit, and C-unit) • Analysis of Speech unit (e.g., independent clause/sub-clausal unit)

14. Qualitative Analysis of Qualitative Data • A range of qualitative data analysis approaches that experimental researchers can consider using to address issues such as why and how participants do or do not behave in a certain manner during an experimental period. • See e.g., Berge 2007; Dörnyei 2007; Duff 2008; Friedman 2012; Johnson & Christensen 2008; Punch 2005; Richards 2003 for options of how qualitative data analysis can be performed).

15. Qualitative Analysis of Qualitative Data • Researchers can perform frequency counts and content analysis (see e.g., Galaczi 2014 who provides useful and clear explanations of content analysis and how it can be done). • Use of anecdotes or direct quotes from the participants • Analytic induction method (see e.g., Kelle 1995) • Miles and Huberman’s (1994) qualitative data analysis framework : data reduction; data display, and drawing and verifying conclusions

16. Discussion • Think about an experimental study you have read so far? What was the research aim? If you could combine qualitative data into the study, what qualitative data technique would you chose? Why? • What do you think could be potential problems or difficulties in combining quantitative and qualitative methods in one experimental study?