Hsien-Chin Liou and Chih-Hsin Lee hcliu@mx.nthu.tw CALICO 2006

1. Another peek into what today’s English learners as researchers actually do in production with the scaffolding of 3 e-referencing tools Hsien-Chin Liou and Chih-Hsin Lee hcliu@mx.nthu.edu.tw CALICO 2006 Foreign Languages and Literature, National Tsing Hua University

2. Overview • Purpose of the study • Literature Review • Research Questions • Data Collection • Data Analysis • Conclusion

3. Purpose of the Study • Recently the pedagogical effectiveness of referencing tools in assisting language learning has attracted quite a lot of attention (e.g., Rundell, 1999; Yang, 2005). • The aim of this study is to investigate the outcome and process of 22 college learners while doing writing and translation tasks. Their preferences and look-up strategies of using 3 web-based referencing tools were documented. The tools were a bilingual dictionary, a bilingual concordancer, and a collocation retrieval program.

4. Literature Review- corpus investigation • Chambers (2005) examined the data of the students' consultation of the corpora, including choice of search word(s), analytical skills, the problems encountered, and their evaluation of the activity. • Although corpora consultation can complement foreign language learning in various educational contexts, limitations were also found such as the small size of corpora and lack of learner training.

5. Literature Review- Dictionary Look-up Behavior • Dictionary look-ups have been demonstrated as useful scaffolds when learners are working on language tasks (Laufer & Hadar,1997; Rundell, 1999). • Factors of word relevance, word inferability and learners’ vocabulary knowledge were shown to influence look-up behavior (Hulstijn,1993). • Inferring ability is related to word consultation in a less straightforward manner than is learners’ existing vocabulary knowledge.

6. Literature Review- Dictionary Look-up Behavior • Rundell (1999) recommended learners to use monolingual dictionaries rather than bilingual dictionaries due to their richer information about syntactic behavior, collocation, word frequency, or synonyms. • The bilingualised dictionary yielded the best results on both comprehension and production (Laufer & Hadar, 1997). • When foreign-language readers look up the meaning of unfamiliar words can be influenced by the language task and learner variables.

7. Literature Review- Dictionary Look-up Behavior • Frankenerg-Garcia (2005) examined how learners used multiple types of reference materials altogether while doing translation. He used a detailed grid for types of query on various tools and found bilingual types were preferred. • Along the same line of Bland, Noblitt, Armington and Gay (1990), Hulstijn (1993), and Liou (2000a), Liou (2000b) claims that the use of online recording programs to do unobtrusive observation of learner behavior can yield valuable data for applied linguists in understanding what learners actually do on tasks.

8. Literature Review- corpus investigation • As technologies advance, online corpus consultation has become popular and proved useful for language learning. • Previous studies have shown that the concordancer is a useful tool to enhance L2 learners’ vocabulary learning (Cobb, Greaves, & Horst, n.d.; Kennedy & Miceli, 2001; Lee & Liou, 2003) and writing (Gaskell & Cobb, 2004; Kennedy & Miceli, 200; Yeh & Yu, 2004; Yoon & Hirvela, 2004).

9. Literature Review- corpus investigation • More research is required to investigate how learners behave while working on different tasks with the help of various e-reference tools. • The effectiveness and students’ preferences of using different on-line reference tools should be addressed with mixed research methods so that a more comprehensive picture can be revealed.

10. Research Questions • Does the online writing practice have an impact on the students’ performance? • What can the computer recording data tell us about the students’ look-ups as a function of tool and task differences? • What are the students’ preferences and perceptions of using three online referencing tools in order to complete translation and writing tasks? What are the benefits and difficulties of using them? • What does the process data based on the analyses of the selected students’ think aloud protocols tell us?

11. The Study: (1) Participants • Twenty-two first-year English majors from an intact class of a public university participated in this study. • Most of them did not have previous experiences in CALL or corpus learning. • Five voluntary subjects out of the 22 subjects joined two think-aloud sections.

12. (2) Tools and Tasks: 3 Web-based Referencing Tools (see writing practice on Candle for more detailed information http://candle.cs.nthu.edu.tw )

13. Bilingual Concordancer: TOTALrecall

14. Bilingual Collocation aid program: TANGO

15. A commercial bilingual dictionary, popular in Taiwan: Dr. eye

16. Figure 1: Illustration of the online writing practice unit

17. Figure 2: Illustration of the online translation unit

18. Figure 3: Illustration of the online writing unit

19. (3) Data Collection

20. (4) Procedure of the Study

21. (5) Data Analyses--Quantitative data analysis (5.1)Scores of Prewriting and Post-writing • Students’ prewriting and postwriting were graded by two experienced raters using the scale of grading the Test of Written English by the Educational Testing Service with the criteria of 1 to 6. Table 2. Comparison of pretest and post writing

22. Quantitative Data Analysis (5.2)Scores of Prewriting and Post-writing • Due to the small number of test takers, we used the nonparametric procedure-Sign Test. Table 3. Comparison of writing performance at 2 time points

23. Quantitative Data Analysis (5.3)Tracking Data—Language Use Table 4. The summary of language used for queries for different tasks It seems that the learners used tools more frequently in translation tasks. The number of tool usein translation is about 1.5 times of it in writing tasks.

24. Quantitative Data Analysis (5.4)Tracking Data- Language Use • Users looked up more entries in Chinese for translation, but more English look-ups appeared in writing. This phenomenon may be related to the nature of the tasks. • In translation tasks, users needed to translate the Chinese text into English so that it was necessary for them to use Chinese to search for the English equivalents. • In writing tasks, the learners needed to confirm what they want to express in English so that the frequency of English mode in writing is higher.

25. Quantitative Data Analysis (5.5)Tracking Data- Tool Use Table 5. The use of different tools in different tasks • The frequency of use of Dr. eye (the dictionary) and TOTALrecall (bilingual concordancer) were considerably higher in both writing and translation tasks. .

26. Quantitative Data Analysis (5.6)Tracking Data- Tool Use • Seven out of the 22 learners preferred the use of the bilingual concordancer (TOTALrecall) than the other two tools, while ten of them showed the preference for the dictionary (Dr. eye), and the rest of them used the dictionary (Dr. eye) and TOTALrecall in similar frequency. • Possible factors that influence individual difference in terms of such preference may include learners’ familiarity towards different tools, the design of the tools and the search results, or usefulness of searched information presented by different tools.

27. Quantitative Data Analysis (5.7)Tracking Data The tracker also recorded students’ use of the on-line referencing tools within one month after the post-writing test. After the experiment, 12 out of 22 students continued to use these tools. Table 6. Students’ use of the tools one month after the experiment

28. The languages of students used one month after the experiment while querying were roughly in the same frequency. Quantitative Data Analysis (5.8)Tracking Data Table 7. The summary of language used for queries one month after the experiment

29. Quantitative Data Analysis (5.9)Evaluation Questionnaire –Usefulness of Tools • Most participants thought that only one single e-referencing tool might not be enough to assist them and using more than one tool is the most useful strategy. Table 8. 22 Participants’ perceptions of usefulness of the 3 tools or their combinations

30. Quantitative Data Analysis (5.10)Evaluation Questionnaire-Usefulness of Tools • Dr.eye is more helpful in learning the meaning of the vocabulary. • TOTALrecall is helpful in learning the usage of vocabulary. TANGO is the second helpful tool, and Dr.eye is the least. • TANGO, for the participants, is a more helpful tool in learning collocation due to its collocation patterns.

31. Quantitative Data Analysis (5.11)Evaluation Questionnaire- Usefulness of Tools • The participants think that using e-referencing tools is more helpful in improving Chinese-English translation skill than English writing skills. • Dr.eye is the most helpful tool in improvement of translation, TOTALrecall the second.

32. Quantitative Data Analysis (5.12)Evaluation Questionnaire- Usefulness of Tools • Participants think that TANGO is the most helpful tool in improvement of English writing skill (which contradicts to what they would prefer to use in the future). • The participants reported that they did not know how to use TANGO because they were not very familiar with the functions of the tool. • More intensive orientation and practice would be the first priority for students to familiarize the functions of the tool.

33. Quantitative Data Analysis (5.12)Evaluation Questionnaire • For future use,students generally hold positive attitudes towards using these e-referencing tools in the future. • Mostly they would like to have more than one e-referencing tools, especially when doing translation tasks.

34. Qualitative Data Analysis (1)Think-aloud Protocols • A coding scheme adopted from Frankenberg-Garcia (2005) and modified was taken up in order to identify and classify their look-up strategies. • The coding scheme includes seven categories: 1. Finding an L2 equivalent; 2. Confirming a hunch; 3. Finding a suitable collocate; 4. Choosing the best alternative or synonym; 5. Checking spelling; 6. Retrieving linguistic info from examples; 7. Looking for part of speech.

35. Qualitative Data Analysis (2)think-aloud protocols • Each individual concept was identified, in terms of each individual problem to be solved in writing or translation no matter how many entries they entered. • Regardless of task difference in writing or translation, students’ look-up behavior falls into three major categories: finding an L2 equivalent (50%), confirming a hunch (19%) and retrieving linguistic information from examples (11%).

36. Qualitative Data Analysis (3)think-aloud protocols • In order to obtain information students needed, they tended to do cross referencing very often, which is also reflected in our tracker data. • Each e-referencing tool provides different types of linguistic information. The information each e-referencing tool provided might not be sufficient. Cross-referencing could help students to test and confirm their hypothesis while searching.

37. Qualitative Data Analysis (4)Think-aloud Protocols • Successful use of strategy was defined as finding the correct information and applying it in writing or translation correctly. • The results show that 89% of these 5 students’ problems were solved and applied successfully.

38. Qualitative Data Analysis (5)think-aloud protocols • Among 89% of successful use of strategy, 3 categories: finding an L2 equivalent (43%), confirming a hunch (18%), and checking spelling (9%) were found to be more often used. • Other strategies that were not used successfully might need to be trained.

39. Triangulation of All the Data (1) • The participants also reported that when having problems in English writing, they would tend to search Dr.eye first, then TOTALreacll. • And Dr.eye and TOTALrecall could usually provide the information they needed. • The data confirmed that they would do initial look-up of perhaps an English equivalent of a Chinese entry first, followed by further consultation of how the word should be used in context using either TOTALrecall or TANGO.

40. Triangulation of All the Data (2) • We observed several different types of problems students had while searching for information: unfamiliarity with tools themselves, or with searching techniques, the small size of corpora in the tools, over-reliance on the tools, and carelessness in searching. • The questionnaire responses also pointed to some difficulties that students encountered while using these e-referencing tools. • Only 22.72% of students perceived no difficulty in using the tools, whereas 36.36% of students felt that they had difficulty in using TANGO and 18.18% of students in using TOTALrecall.

41. Conclusion • The statistical comparison indicated this group made significant improvement (the learning products) perhaps through the design and the use of the online writing practice tasks. • It seems that online writing practice with scaffolds of referencing tools can improve their production as evidenced in the learning product data.

42. Conclusion • The acceptance of the innovative corpus tools for references is satisfactory to the learners based on the questionnaire data; definitely more orientation with practice from English teachers is needed for students to acquire mature look-up skills. • Although succinct task differences between writing and translation were not observed, they warrant more future research.

43. Acknowledgements • The paper was funded by a National Science Council project (under the number of NSC 94-2524-S007-001, the CANDLE project). • We also thank Hsin-Ping Yu for programming and sorting the tracked data, Jany Lin for helping collecting and analyzing the data, and Ashley Ho for serving as another rater for grading the writing and coding the think-aloud protocols. • The 22 participating students are also acknowledged.

44. Thank You for Your Attention!