1 / 20

Developing a task-based assessment of EAP pragmatics

Developing a task-based assessment of EAP pragmatics. Soo Jung Youn (soojung@hawaii.edu) University of Hawai‘i at M ā noa 3 rd International Conference on Task-Based Language Teaching Lancaster University, UK.

gwen
Download Presentation

Developing a task-based assessment of EAP pragmatics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing a task-based assessment of EAP pragmatics Soo Jung Youn (soojung@hawaii.edu)University of Hawai‘i at Mānoa 3rd International Conference on Task-Based Language Teaching Lancaster University, UK

  2. Investigating EAP Pragmatics Pragmatics Even with high L2 proficiency, L2 are in great need of L2 pragmatics instruction Advising sessions, e-mails to professor, writing tutorials (Bardovi-Harlig & Hartford, 1990, 1996, 2005) A longitudinal case study of e-mail literacy dvlpmt (Chen, 2006) “the study of language from the point of view of the users, especially of the choices they make, the constraints they encounter in using language in social interaction, and the effects their use of language has on the other participants in the act of communication” (Crystal, 1997, p. 301, italics added) little attention on systematic investigation of learning needs + pedagogic & assessment tasks “referring to nonnative speakers’ comprehension and production of speech acts, and how that L2-related speech act knowledge is acquired” (Kasper & Dahl, 1991, p. 216)

  3. Assessment of Pragmatics • Development of six prototype pragmatic measures and rating scales (Hudson, Detmer, & Brown, 1992, 1995) • Development of general pragmatic tests (Roever, 2005, 2006; Liu, 2007; Tada, 2005) • Remaining Concerns: • how the test batteries are being used and their pedagogical value • not fully aligned with pragmatic needs of a specific context • How can these concerns be addressed?

  4. Task-based Language Assessment • Fundamental Question: • Why and how are task-based assessment being used in particular L2 educational contexts? (Norris, 2002, 2008) • Deal with complex, integrative, task-specific needs • Examples of empirical TBLA studies University-level second & foreign TBLA (Norris et al., 1998; Brown et al., 2002) DSL in Belgium for high-stakes decisions with summative uses of TBLA (Gysen & Van Avermaet, 2005) Reform an entire foreign language curriculum - both pedagogical & assessment (Byrnes, 2002) • TBLA • Provide learning • opportunities • Evaluate • effectiveness • Rethink • the program • …

  5. Study Purpose & Context • To engage in the process of identifying EAP pragmatic needs, specifying intended use of assessment tasks, developing assessment tasks & rating scales • EAP program context: - English Language Institute (ELI) at the University of Hawai‘i at Mānoa - provides academic English instruction for international & immigrant & (U.S. citizens) – in need of assistance with EAP - academic skill-based curriculum

  6. Research Questions • What are major EAP pragmatic needs as perceived by students, instructors, and the program? • What are the intended uses of assessments of EAP pragmatics within a university EAP program? • What kinds of assessment tasks and rating criteria can be developed to meet both the EAP pragmatic learning needs and intended assessment uses in the target program?

  7. Study Procedure 1) Needs analysis 2) A specification of intended uses of EAP pragmatic assessment 3) Development of pragmatic learning outcomes proposal 4) Development of pragmatic target tasks 5) Development and validation of EAP pragmatic assessment tasks and task-dependent rating scales

  8. Needs Analysis - Procedure Semi-structured interviews Questionnaire 12 ELI Students a) Focus group b) Interviews 20 EAP pragmatics-related situations 102 ELI Students (Grad/UG: 43.1%/ 56.9%) synthesis 7 ELI Instructors 3 ELI Administrators

  9. Questionnaire Main Findings • 1(not at all necessary)-2-3-4 (very necessary) scale: “learning need” • Cronbach alpha = 0.9 • Lowest mean = 2.9 (out of 4) • Five most needed tasks (by students) 1. Refuse politely to professor’s request 2. Request a recommendation letter 3. Write a cover letter to apply for a job 4. Write an appropriate e-mail to professor 5. Ask clarifying questions to professors appropriately

  10. Any gaps among informants? • Do all informants mention same needs? • Among the least needed tasks by students.. But, mentioned and emphasized by ELI teachers!! Know how to appropriately comment, criticize, and compliment on classmates’ in class work

  11. Graduate vs. Undergraduate Students Recommendation letter request Write an e-mail Write a cover letter

  12. Across Different Levels Recommendation letter request Write a cover letter

  13. Intended Uses of EAP Pragmatic Assessment • Who use? • ELI students • ELI instructors • Prospective instructors • & students • What is being • assessed? • Overall pragmatic ability with • a range of EAP tasks that • place common pragmatic • demands on students • Why • assessment? • Diagnose • Self-assessment • Pedagogical tasks • Measure students’ learning • progress and achievement • Teaching materials • Who/What is • being impacted? • Not intended to reform, but • strengthen the program • Satisfaction of needs • Evaluate effectiveness • Professional development • Raising awareness

  14. Assessment Tasks 2. Write an e-mail to a potential employer to send application packet 3. Write an e-mail to refuse professor’s request based on identified target tasks: • Write a • recommendation • request e-mail • to professor • Task 1 4. Write constructive comments on cover letter written by classmate 7. Role-play with a professor in situations of making requests and refusal 5. Give oral peer-feedback on classmate’s request e-mail to professor 6. Role-play with a classmate in situations of making suggestions and disagreement

  15. Task-Dependent Rating Criteria • Analytical rating criteria with detailed descriptions • Each task has different rating criteria • Feedback from domain experts • 3 (good) – 2 (able) – 1 (inadequate) • e.g.) 1. Write a recommendation letter request • e-mail to professor • Task rating criteria

  16. Procedure Examinee Participation Rating • 3 raters • (experienced ESL/EFL teachers with MA in ESL degrees) • Had three consecutive separate training sessions • Asked to keep monitoring, to document any difficulties, reasoning of rating • 40 students participated • (low – high proficiency) • Individually completed all 7 tasks • (about 1 hour to 1 hour 15 min) • Either use a computer or hand-write • Role-play audio-recorded

  17. FACETS summary Write a constructive comment on cover ltr written by a classmate most difficult 0.90 easiest Role-play with a professor (request, refusal)

  18. Concluding Marks - so what? • Ongoing process • Future investigations (e.g., how these assessment are actually put to use, helpful to teachers & learners?, impact on the program) • Need more empirical studies(e.g., do actually assessment tasks promote L2 pragmatic learning? If so, how?) • Rethink roles of assessment

  19. Mahalo! (soojung@hawaii.edu)Sincere appreciation to:Dr. John M. Norris& ELI, HELP, Research ParticipantsFunded by University of Hawai‘i at Mānoa Art & Sciences Student Research AwardGraduate Student Organization Grant

  20. Reference Bardovi-Harlig, K., & Hartford, B. S. (1990). Congruence in native and nonnative conversations: Status balance in the academic advising session, Language Learning, 40, 467-501. Bardovi-Harlig, K., & Hartford, B. S. (1996). Input in an institutional setting, Studies in Second Language Acquisition, 18, 171-188. Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, New Jersey: Lawrence Erlbaum Associates. Brown, J. D., Hudson, T., Norris, J., & Bonk, W. J. (2002). An investigation of second language task-based performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center. Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19, 419-437. Chen, C.-F. E. (2006) The development of e-mail literacy: From writing to peers to writing to authority figures. Language Learning & Technology, 10(2), 35-55. Gysen, S., & Van Avermaet, P. (2005). Issues in functional language performance assessment: The case of the certificate Dutch as a foreign language, Language Assessment Quarterly, 2, 51-68. Hudson, T., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center. Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics (Technical Report #7). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center. Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391-415. Norris, J. M. (2000). Purposeful language assessment. English Teaching Forum, 38(1), 18-23. Norris, J. M. (2002). Interpretations, intended uses and designs in task-based language assessment, Language Testing, 19, 337-346. Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Technical Report #18. Honolulu, HI: Second Language Teaching and Curriculum Center. Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229-256.

More Related