Effects of Web-based Oral Activities Enhanced by Automatic Speech Recognition

1. Effects of Web-based Oral Activities Enhanced by Automatic Speech Recognition Tsuo-Lin Chiu, Hsien-Chin Liou, Yuli Yeh National Tsing Hua University Taiwan R.O.C.

2. Overview • I. Literature review • II. CandleTalk • III. Methodology of the evaluation study • IV. Results and discussion • V. Implications and conclusion

3. I. Lit review: Communicative Competence • Communicative competence has been the goals for the teaching of English as a second language and as a foreign language • One of the components of communicative competence by Canale and Swain (1980) is sociolinguistic competence

4. Speech Act • Speech acts (Austin, 1962; Searle, 1969) are the units of words that speakers use for achieving communicative functions • Cohen and Olshtain (1993): explicit teaching of speech acts could help benefit in accelerating the process of developing communicative competence • Cohen (1994): two abilities for explaining successful speech act performance • Sociolinguistic ability: selection of proper linguistic forms for speech act performance • Sociocultural ability: selection of proper speech acts based on sociocultural factors

5. A recent CALL application is Automatic Speech Recognition (ASR) technology. • 4 Types of ASR Recognizers (Wachowicz and Scott, 1999): • Speaker-independent ASR systems • Speaker-dependent ASR systems • Discrete word ASR systems • Continuous ASR systems

6. ASR Advantages • For teaching oral skills…. • Being a private tutor (for speaking) • Simulated face-to-face conversation • Face-saving • Freedom of practice • Suitable for assisting those who rarely have chances to speak with native speakers such as EFL contexts in Taiwan

7. ASR Limitations • Recognition Errors • Give false feedback

8. Review of ASR Programs • MY English Tutor (MY ET) (,Chen & Chiu, 2005; Tsai, 2003) • Subarashii (Bernstein et al., 1999) • Virtual Conversations (Harless, Zier, & Duncan, 1999) • The Voice Interactive Training System (VILTS) (Rypa & Price, 1999) • The FLUENCY (Eskenazi, 1999) • Project LISTEN’s Reading Tutor (Mostow & Gregory, 1999) • NTNU pronunciation site (Chen, 2005)

9. Reviews of the literature showed that most studies on ASR were description of the system, pilot testingon a small number of subjects, or relying on learners’ self-report as measures of the effectiveness of environments. • There seems to lack empirical data from a larger scale of testing that examines the effectiveness of ASR on EFL learners’ language learning (cf. Zhao, 2003) .

10. Purposes of the study • Therefore, the current study: • presented the environment of CandleTalk • examined learners’ perceptions of CandleTalkand its effectiveness on teaching target language skills.

11. Research Questions • What is students’ perceptions toward a self-developed system, CandleTalk? • Will the application of ASR technology in CandleTalk improve EFL students’ oral production by using appropriate and intelligible utterances?

12. A self-devised ASR system, CandleTalk (http://candle.cs.nthu.edu.tw) • Based on dialogues with content on Tawain’s culture • Contents of Four Units • Unit 1: greeting and parting, famous singer • Unit 2: requesting, local resort • Unit 3: complaining and apologizing, online games • Unit 4: complimenting, night market

13. An example of Flowchart in CandleTalk (unit1)

14. Introduction Page

15. Selection of voices

16. Main page

17. Review Page

18. Dialogue Record

19. Methodology • Participants • 49 college freshmen from a public university • 29 English majors, 20 non-English majors • Instruments • Online Questionnaire • Using 5-point Likert Scale • Scores were given (Strongly agree=5, A=4, N=3, D=2, SD=1) • A spoken version of Discourse Completion Test (DCT) as pre- and posttest

20. Example of DCT • An example of DCT test: • “You are going to go abroad for a week. You are worried that nobody can feed your cat while you are abroad. You are going to ask your friend to take care of your cat. What would you say?” • A sample answer might be: • “Chris, I am going to Japan this week. I was wondering if you could help me take care of my cat?”

21. Data collection procedures

22. Data Analysis • The results of the questionnaire were coded according to the categorizes of items: • Results of DCT were rated based on two criteria: • Comprehensibility of the speech. • Use of speech acts.

23. Evaluation Questionnaire • Results of evaluation questionnaire were analyzed and ranked based on three categories: • Items concerning the 4 units • Functionality of the site • Overall effectiveness of CandleTalk

24. Items of the 4 units

25. Functionality of the site

26. Overall Effectiveness

27. Major Results from Questionnaires • The first research question is well-supported by the results of the questionnaire that students in general held positive attitude toward CandleTalk regarding its pedagogical purposes, designing features, and interface • The recognition rate and the recording interface require improvement

28. Rating of DCT Responses • A total of ten questions (with different orders in the pre- and posttest) • An English native-speaker professor helped with range finders by listening to 20 samples (49x2x10), 10 recorded sound files from pretest and 10 from posttest • The data were rated by two raters who are both MA TEFL students, disagreements solved by discussion with interrater reliability of .87 • Full score of the test is 50 (3x10 + 2x10)

29. Rating of DCT Responses • Comprehensibility (3-point scale,) is about how easy to understand an utterance (Munro & Derwin, 1995) • 0 point = totally incomprehensible utterance • 1 point = utterance which contains poor pronunciation or is difficult for comprehension • 2 point = utterance which is fully comprehensible • Use of Speech acts (4-point scale)

30. Rating scale of Use of Speech Acts

31. Comparison of pre- and posttest • Paired T-Test was used to see whether there are significant differences in scores • Results were analyzed from three categories: • Total score (50=20+30) • Comprehensibility (full score=20) • Use of speech act (full score=30)

32. Comparison of pre- and posttest (total score)

33. Comparison of pre- and posttest (comprehensibility)

34. Comparison of pre-and posttest (use of speech act)

35. Comparison of two groups (total score) *p< .05

36. Comparison of two groups (comprehensibility) *p< .05

37. Comparison of two groups (use of speech act) *p< .05

38. Major Results From the Comparison of DCT Results • Students showed significant increase in total score and use of speech act, but not in comprehensibility • CandleTalk is effective in helping students how to perform the six speech acts appropriately, but it did not significantly help the students improve the clarity of their speech

39. Major Results From the Comparison of DCT Results • When group differences were concerned, the non-FL group showed significant improvement in the categories of total scores and use of speech act, but the FL group did not show any significant improvement in any of the three categories • The pretest scores of non-FL group were lower than those of FL group, but in the posttest non-FL group caught up.

40. Answer to the second research question: • the students did improve their oral skills in terms of the use of speech acts, but the salient improvement was mostly made by the non-FL group, which is also the relatively lower proficiency group

41. Students’ perceptions toward the environment • Students were positive toward CandleTalk, and believed it met their expectation of learning English • The results consistent with previous studies (Harless, et al., 1999, Hsia, et al., 2004, and Tsia, 2003) • Use of native culture as topics of dialogues (Savignon & Sysoyev, 2002) and the feature of simulated real-life conversation (e.g. Bernstein, et al., 1999, Wachowicz & Scott, 1999) seem helpful.

42. Students’ perceptions toward the environment • The value of feedback given by the speech recognizer (e.g., Eskenazi, 1999) • The recognition rate/accuracy • The recording interface design

43. Students’ learning outcome after the instruction • An ASR-supported learning environment, like CandleTalk, is effective on enhancing the target language skills • The result is similar to Harless, et al, (1999) in which pedagogy and ASR technology combined could enhance or sustain the language skills • Students of lower language proficiency seem to benefit more from the instruction supported by ASR • The result is similar to Tsia (2003) that it is the lower level group benefited more form the application of speech recognition technology • Limited improvement of the FL group

44. Development Implications for an ASR supported Conversation Environment • Innovative formats of dialogue presentations are needed • The interface design should be user-friendly • The input provided by the environment could be diverse and relevant to learners

45. Development Implications for an ASR supported Conversation Environment • The feedback given by the recognizer should be a reliable and helpful indicator for learners’ performances in speaking • It takes the collaboration between the subject matter experts and the programmers to construct a better recognizer and further explore the potentials of ASR on language learning

46. Pedagogical Implications • Instructional materials supported with ASR could be an effective alternative for teaching and learning of speaking • The explicit teaching of use of speech acts would help facilitate EFL learners’ development of communicative competence • Clear orientations and directions during CALL instruction are still needed

47. Limitations of the Study • No control group • The use of Discourse Completion Test

48. Suggestions for Future Research • To explore the application of ASR on other aspects of oral skills • To investigate the effectiveness of ASR on learners in other contexts, of different ages, and of different proficiency levels

49. The End Thank you for your attention! Hsien-Chin Liou hcliu@mx.nthu.edu.tw Tsuo-Lin Chiu g925256@alumni.nthu.edu.tw This project, Naitonal Science and Technology project for E-learning, under the grant number NSC 92-2524-S007-002, is supported by National Science Council, Taiwan