Intelligent CALL (ICALL)

1. Integrating ICALL into synchronous CMCCan intelligent computer-generated feedback be provided in learner-to-learner interactions? Markus Dickinson, Soojeong Eom, Natalia Jacobsen, Yunkyoung Kang, Chong Min Lee, Ken Petersen, & Rebecca Sachs iicall2 conference; December 8, 2007; Waterloo

2. Intelligent CALL (ICALL) • ICALL provides many potential means of facilitating L2 development • Generation of detailed information regarding sources of learner errors • Precise feedback on learner errors • Can be tailored to learners’ common mistakes, activity goals, proficiency levels, abilities, etc. • Fosters awareness of relevant language forms • Tracking of improvement across exercises, using learner models (cf. Amaral & Meurers 2006, Heift & Schulze 2007)

3. Intelligent CALL: Limitations • However, there is a tension between • Contextualized language use is increasingly emphasized, and with increasing success (e.g., Amaral et al. 2006; Amaral & Meurers 2006)… • But actual communicative interaction is relatively unexplored (though see Petersen 2006) • In order to manage computational complexity: • Exercises are often restricted to the sentence level • Activities often do not simulate true communication The ability of an ICALL system to provide meaningful, accurate feedback The flexibility an ICALL system allows for in terms of meaningful, communicative interaction

4. Synchronous CMC • Synchronous computer-mediated communication (CMC) between L2 learners can provide a beneficial context for L2 development • Focus on meaningful communication • Activities can approximate ‘target tasks’ relevant to real-life communicative situations, with learners functioning as language users (Ellis 2003) • Possibly optimal psycholinguistic environment for learners to negotiate for meaning and make form-meaning connections (Blake 2000, Doughty & Long 2003) • Less pressing time constraints and less ephemeral language (e.g., through possibility of re-reading) in the written modality • Reduced processing demands, which may serve as an equalizer for learners with lower working memory capacity (Payne & Whitney 2002) • Attention to form and monitoring of language production (Chapelle 2003, Pellettieri 2000) • Use of more complex language (Warschauer 1996)

5. Synchronous CMC: Limitations • Concerns regarding the quality of learner-learner interactions with respect to potential benefits for learning • The blind leading the blind? • Without feedback from a trusted authority, learners might reinforce each other’s errors (Kern 1995) and/or simply not have the resources necessary for correcting each other (Blake 2000) • In negotiating for meaning, learners may naturally tend to focus on lexis without attending to morphosyntax (Blake 2000) • Learners might not be concerned with grammatical accuracy (Kern 1995), in which case available planning time is a moot point Teachers may need to set explicit expectations for grammatically correct language, while simultaneously trying to balance this with a primary focus on meaningful communication (Lee 2001, Pellettieri 2000)

6. Combining the benefits • Can ICALL and synchronous CMC be integrated in a way that exploits the unique benefits of each while avoiding their limitations? detailed, informative, individualized feedback communicative purpose negotiation and interactional modifications attention to form and meaning approximation of real-world target tasks with less pressing time constraints development of functional L2 proficiency with grammatical accuracy Intelligent computer-generated feedback in synchronous, task-based, computer-mediated learner-learner interactions

7. A balancing act • We want learners’ interactions to be as free as possible, promoting authentic communication • At the same time, we are restricted by limitations of ICALL processing and must constrain the situation somehow (cf. discussion in Amaral et al. 2006) Guided picture-based information-gap tasks can constrain the vocabulary and domain in ways that: • reduce many of the complexities involved in generating feedback (computationally speaking) • still allow for communicative interaction between learners (interactional authenticity if not real-world authenticity)

8. Participants • English-speaking university students in first-year Korean classes Can (or should) beginning learners use CMC? • Important for learners to develop the ability to use language to communicate meaning as opposed to simply ‘displaying’ language or manipulating L2 forms systematically without attending to meaning • However, unconstrained communicative tasks might be stressful (or a long shot) for beginning learners • We hope to design our tasks in such a way that participants will be comfortable communicating meaningfully in the L2 • With appropriate guidance • Within a constrained and familiar environment

9. A communicative task we’re considering: Spot the differences • Each participant sees one version of a house and must exchange information in the L2 in order to find similarities and differences between the two versions • Not so authentic in terms of real-world relevance, but ‘interactionally authentic’ – the sort of task often used in interaction research to target specific areas of language and promote negotiation and L2 learning • To increase motivation, set up as a competition between pairs of participants • Record (in a provided chart) the activities and locations of all characters in partner’s house • Indicate whether each of these represents a similarity or a difference between the pictures • Compare scores at the end to those of other participant pairs

10. Spot-the-differences task (picture 1)

11. Spot-the-differences task (picture 2)

12. Target of feedback: Korean particles • In Korean (as in Japanese, cf. Nagata 1995), postpositional particles are used to indicate grammatical functions and thematic roles (e.g., who is doing what to whom). • They must be used even in simple sentences. • Particles are taught from the beginning of L2 Korean study, but: • The system is quite complex and difficult to master for adult learners of Korean • Particle errors account for a substantial proportion of the mistakes made by beginning learners (Ko et al. 2004) • Errors persist even at advanced levels

13. 에(‘e’) 에게(‘ege’) English ‘to’ 께(‘kke’) 한테(‘hante’) Difficulties with Korean particles • No one-to-one correspondence between Korean locative particles and English prepositions • Korean locative particles mark distinctions not made in English Location of a static object 에(‘e’) Location of a dynamic activity 에서(‘eseo’) English ‘in’ Location an action moves through 을/를(‘eul/reul’)* (*also an object marker)

14. Examples of targetlike and non-targetlike Korean particle use • What is in the kitchen? • 주방-에 뭐-가 있어요? 주방-에 뭐-*를있어요? kitchen-LOC what-SUBJ is kitchen-LOC what-OBJ is • Father is grilling meat. • 아버지-가 고기-를 구어요. 아버지-가 고기-*가굽고 있다. father-SUBJ meat-OBJ grill father-SUBJ meat-SUBJ is grilling • A cat is in the living room. • 고양이-가 거실-에 있어요. 고양이-가 거실-*에서 있다. cat-SUBJ living room-LOC is cat-SUBJ living room-LOC is (TL examples are on the left with particles in green; non-TL are on the right with particles in red)

15. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

16. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … TASK PICTURE: Partners have slightly different versions and must communicate to find differences. They can scroll over the picture to enlarge it. S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

17. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles WORD & PARTICLE BANKS: To create a sentence, participants click on words and particles… 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

18. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … …which then appear in the sentence drafting area. S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

19. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … If they want help with Korean particle usage, they can request feedback on their sentences before entering them into the conversation. S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

20. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles FEEDBACK AREA:Here, participants receive metalinguistic feedback with advice on particle usage. 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

21. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … S D S D When they are ready, they click SEND to enter their utterance into the conversation. 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

22. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … S D S D CHAT WINDOW:They can scroll up and down to review the conversation so far. 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

23. [ParticipantA]: 오빠가 어디에 있어요? [ParticipantB]: 거실에 있어요. [ParticipantA]: 오빠가 거실에서 뭐해요? [ParticipantB]: … GAME RECORD:When participants find similarities or differences, they drag the relevant words for locations and activities here to record information about their partners’ pictures, then click on ‘S’ or ‘D’ to indicate whether the pictures match in those respects or not. S D S D 오빠가 소파를 책을 읽어요. S D S D S D S D CHECK SEND S D S D In your sentence, 소파is marked with the particle 를, which suggests that 소파is an object. Instead, you need the particle에서attached to 소파in order to indicate that 소파is the location of a dynamic activity. Word Bank Particles 침대 화장실 에 는 이 에서 가 의자 거실 을 … 읽었어요 책 울었어요 소파

24. The importance of instructions • Piloting so far (with native speakers of Korean in face-to-face interactions) has suggested that carefully worded instructions will likely be needed to: • Encourage participants to use complete sentences • Emphasize that the characters in the pictures are members of the same family (so that terms such as ‘mother’, ‘brother’, etc., can be used naturally) • Clue participants in to the fact that differences will involve the locations and activities of family members, as opposed to characteristics of furniture (for example)

25. From activity to processing • How can we process CMC as input?  Built-in constraints • Intensive feedback is provided on one particular error type • Learners’ construction of sentences is guided by… • The nature of the picture-based task (constrains vocabulary) • Instructions and the game record chart • Word and particle banks, which limit the types of argument structure by limiting the verbs that can be used, and which moreover… + May be necessary for beginning learners who can’t type in Korean + May serve as a scaffold for using receptive vocab in conversation • Upshot: Processing can focus just on detecting particle errors in a known domain.

26. Intensive feedback • Some have argued that intensive feedback may be more effective in certain contexts than wide-ranging incidental feedback on a variety of error types (e.g., Lyster 1998, Nicholas, Lightbown, & Spada 2001) • Intensive feedback = focusing consistently on one pre-selected error type • Meta-analyses have not found significant differences, but too few primary studies to tell for sure; in any case, both produce large effect sizes (e.g., Mackey & Goo 2007, Russell & Spada 2006) • In our study, we will inform learners that they will only be receiving feedback on particles • Important for meaning (i.e., communicating who is doing what to whom) in Korean • Will hopefully prevent learners from mistaking non-feedback for correctness • Leaves open the possibility of providing other feedback, if needed

27. Meaningful communication? • A spot-the-differences task provides a possible forum for learners to use an L2 in purposeful, communicative ways, but… • How much will the participants focus on meaningful communication if it’s clear that the feedback is focusing exclusively on Korean particles? What else can we do to encourage a balance between focusing on intended meanings and focusing on the forms required to express those meanings accurately? • How free will the participants be to express a creative range of ideas in the context of this task? What can we do to make the word bank sufficiently rich for their communicative purposes? • How fluid will communication feel if participants have to take the time to search for each word they want to use? What can we do to make the words in the bank as easy to access as possible?

28. Future directions • Pilot the game with L2 learners • Get a clearer sense of what to expect in learner input • Test how the word bank actually plays out • Develop the system • Activity, error diagnosis, and feedback modules • Processing techniques • Make the tasks more complex, meaningful, and relevant to real-life communicative situations • Create realistic rationales for having to find differences (e.g., a detective and witness discussing a crime scene as it appears just before and after a crime) • Explore other sorts of picture-based information-gap tasks • Remove or expand the word bank • This would require additional constraints, however, to keep processing feasible • Develop activities to target more constructions

29. Opportunity to test techniques • A combination of techniques will ultimately be used. • Linguistic processing will be kept separate from error diagnosis and feedback generation. • We can use this setting to experiment with different processing technologies • More traditional, anticipation-based regular expressions • Basic linguistic abstraction of POS tags and chunks to provide annotation of input (cf. Amaral & Meurers 2006) • Experimental ill-formed input checking • Use mismatches in 2 different parsers to detect particle errors (cf. Metcalf & Boyd 2006) • One parser captures particle usage patterns from real language • One parser captures general argument structure patterns between words, irrespective of particles

30. Opportunity to test questions of SLA theory and language pedagogy • We will be able to explore learners’ uses of various types of feedback in CMC, using pre-test / treatment / post-test designs to assess L2 development • Intensive vs. non-intensive feedback? • Metalinguistic information vs. prompts vs. recasts? (etc.) • Content-oriented feedback? • Eventually, we can use such a set-up to investigate ways of optimizing feedback: • For different areas of language • For learners of different proficiency levels • For learners with different aptitude profiles • Using and integrating this system with the Korean language curriculum at Georgetown

31. Questions or comments? Please email:

32. References • Amaral, L., & Meurers, D. (2006). Where does ICALL fit into foreign language teaching. CALICO 2006. University of Hawai’i. • Amaral, L., Metcalf , V., & Meurers, D. (2006). Language awareness through re-use of NLP technology. Pre-conference Workshop on NLP in CALL – Computational and Linguistic Challenges. CALICO 2006. University of Hawai’i. • Blake, R. (2000). Computer-mediated communication: A window on L2 Spanish interlanguage. Language Learning & Technology, 4(1), 120-136. • Chapelle, C. (2003). English language learning and technology. Philadelphia: John Benjamins. • Doughty, C.J., & Long, M.H. (2003). Optimal psycholinguistic environments for distance foreign language learning. Language Learning & Technology, 7(3), 50-80. • Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford University Press. • Heift, T., & Schulze, M. (2007). Errors and intelligence in computer-assisted language learning: Parsers and pedagogues. Routledge.

33. References • Kern, R. G. (1995). Restructuring classroom interaction with networked computers: Effects on quantity and characteristics of language production. The Modern Language Journal, 79, 457-473. • Ko, S., Kim, M., Kim, J., Seo, S., Chung, H., & Han, S. (2004). An analysis of Korean learner corpora and errors. Hanguk Publishing Co. • Lee, L. (2001). Online interaction: Negotiation of meaning and strategies used among learners of Spanish. ReCALL Journal, 13(2), 232-244. • Lyster, R. (1998). Form in immersion classroom discourse: In or out of focus? Canadian Journal of Applied Linguistics, 1, 53-82. • Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction and second language acquisition: A series of empirical studies (pp. 407-452). Oxford: Oxford University Press. • Metcalf, V., & Boyd, A. (2006, December). Head-lexicalized PCFGs for verb subcategorization error diagnosis in ICALL. Workshop on Interfaces of Intelligent Computer-Assisted Language Learning. The Ohio State University.

34. References • Nagata, N. (1995). An effective application of natural language processing in second language instruction. CALICO Journal, 13(1), 47-67. • Nicholas, H., Lightbown, P. M., & Spada, N. (2001). Recasts as feedback to language learners. Language Learning, 51, 719-758. • Payne, J. S., & Whitney, P. J. (2002). Developing L2 oral proficiency through synchronous CMC: Output, working memory, and interlanguage development. CALICO Journal, 20(1), 7-32. • Pellettieri, J. (2000). Negotiation in cyberspace: The role of chatting in the development of grammatical competence in the virtual foreign language classroom. In M. Warschauer & R. Kern (eds.), Network-based language teaching: Concepts and practice, 59-86. Cambridge: CUP. • Petersen, K. (2006, December). Measuring L2 development in an ICALL context. Workshop on Interfaces of Intelligent Computer-Assisted Language Learning. The Ohio State University. • Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching. Amsterdam: John Benjamins. • Warschauer, M. (1996). Comparing face-to-face and electronic discussion in the second language classroom. CALICO Journal, 13(2), 7-26.