Assessing Academic Writing with a Pragmatic Email Task

1. Assessing Academic Writing with a Pragmatic Email Task Randall Rebman Northern Arizona University

2. Literature Review • Pragmatics are part of most notable models of communicative competence (Bachman & Palmer, 2010; Canale, 1983; Canale & Swain, 1980) • The testing of second language competence is an underexplored area in second language assessment (Roever, 2011)

3. Target Domain • Sample of writing tasks outlined by Grabe & Kaplan’s (1996) taxonomy • Notes and memoranda • Lecture notes, reports (expository) • Letters including speech acts like refusals, requests and recommendations • Recounts & Narratives • Argumentative Essays

4. Rationale For Task Choice of Tasks Building on the Writing Framework for the TOEFLdeveloped by Cumming et al., (2000) this test domain is made up of three tasks: • Independent invention task • Interdependent tasks • Interdependent situation-based task

5. Purpose of Test Development • To place L2 students in different levels of writing ability. • To decide if students matriculate into the university from the intensive English program. • To include a representation of writing tasks that second language writers will be required to produce in university contexts (Bridgeman & Carlson, 1983).

6. Why an Email Task? • In our own classes we see students struggle with using the proper conventions of email for communication. • There is a potential for positive washback (Crusan, 2010)in that teachers may begin to prioritize teaching the register and genre features of emails. • Some writing assessment researchers have argued for more situation-based writing tasks (Cumming et al., 2000), but such tasks need prototyped for local contexts (Weigle, 2002). • Adding an email task to a writing test can expand the range of the construct of academic writing that is assessed and provide an additional writing sample.

7. Limited Test Domain Integrated Task: Summary of a chart Independent Task: Prompt-based Argumentative essay Situational-based Task: Request to a Professor

8. Research Questions for Test Trialing of Email Task #1 • Can the same rater produce consistent ratings of an email writing task using a new rubric? • Is the email response task testing academic writing ability in a different way than the integrated and independent writing tasks?

9. Research Questions for Email Task Test Trialing #2 • Can different raters produce consistent ratings of an email writing task using a new rubric? • Is the email response task testing academic writing ability in a different way than the integrated and independent writing tasks?

10. Test Trialing #1 & #2 Participants • Trial #1: n= 174 • Trial #2: n= 103 • International students mainly representing China, Saudi Arabia, Japan, Korea, and Kuwait • Ages ranged from 18-24 • All were pre-university students required to take the English placement test to determine level placement in the Program of Intensive English or advancement to the university

11. Methods • A holistic scale for prototype email task was created using the empirical method (Weigle, 2002) • Quantification: a 6-point rubric operationalized the construct of writing ability on the email task which resulted in a score of 0-5 given by a single rater

12. Methods • A Spearman-Brown correlation coefficient is used to measure intra-rater reliability and inter-rater reliability for RQ1. • A Spearman-Brown correlation coefficient is used to measure the internal consistency between the email task, integrated task and independent task RQ 2. • Spearman-Brown was chosen over Pearson Coefficient because the data is not truly continuous (Hatch & Lazerton, 1990).

13. Methods Email task scale scoring criteria • Language use • Grammatical and lexical features • Register awareness, including appropriate forms of address • Genre markers specific to emails • Topical relevance • Task completion

14. Task Characteristics • Task Prompt Directions (3 minutes): Read the question below. Plan, write, and revise an email. Use the space below to prepare writing your email. You may begin now. Question: You are new at XXX University. You do not know what classes to take. Write an email to Professor Smith to do the following: 1) introduce yourself 2) explain your problem 3) ask for advice

15. Results of Test Trial#1 Descriptive Statistics Note. CI = confidence interval; LL = lower limit; UL = upper limit Correlation Coefficient for Intra-rater reliability Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;

16. Results of Test Trial#1 Descriptive Statistics Across Writing Tasks Note. CI = confidence interval; LL = lower limit; UL = upper limit Correlations Between Writing Tasks Note. df = 172; alpha .05; rho critical = .364.

17. Results of Test Trial#2 Descriptive Statistics Note. CI = confidence interval; LL = lower limit; UL = upper limit Correlation Coefficient for Inter-rater reliability Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;

18. Results of Test Trial#2 Note. CI = confidence interval; LL = lower limit; UL = upper limit Correlations Between Writing Tasks Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;

19. Discussion • Students had higher mean scores on the email task than on the other two task types for test trail#2 • What does this mean for implementing the new task? • The dispersion of test scores did not distinguish students by writing ability. • The task appears to be too simple or the scale made it too easy to get a high score. • There is also the possibility that the different test takers in test trial #2 were more familiar with the conventions of an email than those in test trial #1

20. Implications • The email task could be improved by adding complexity to the task design. • This could be done by giving more input for learners to respond to, such as a sample email from a professor to which they must respond to • The scale must be revised to better distinguish criteria expected for different bands of rubric • A sample of emails to faculty members could be gathered to identify pragmatic features lacking in current task design • Future research needs to determine if the responses to an email task produces different textual features.

21. References Bachman, L. F., & Palmer, A. (2010). Language assessment in practice. Oxford: Oxford University Press. Bridgeman, B., & Carlson, S. (1983). Survey of academic writing tasks required of graduate and undergraduate students. TOEFL Research Report 15. Princeton, NJ: Educational Testing Service. Canale, M. (1983). From communicative competence to communicative language pedagogy. In J. Richards & R. Schmidt (Eds.), Language and communication (pp. 2–27). London: Longman. Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1, 1–47. Crusan, D. (2010). Assessment in the Second Language Writing Classroom. Ann Arbor: University of Michigan Press. Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework: A working paper. (TOEFL Monograph Series Report No. 18). Princeton, NJ: Educational Testing Service. Grabe, W., & Kaplan, R. (1996). Theory and practice of writing. New York: Longman. Hatch, E., & Lazaraton, A. (1990) .The research manual: Design and statistics for applied linguistics. Boston: Heinle & Heinle Publishers. Weigle, S. C. (2002). Assessing Writing. Cambridge: Cambridge University Press Roever, C. (2011). Testing of second language pragmatics: past and future. Language Testing, 28(4) , 463–481.