290 likes | 520 Views
The presentation structure
E N D
1. Re-examining Factors that Affect Task Difficulty in TBLA Shaoqian Sheila Luo
English Department, CUHK
English Department, BNU, China
Supervisor: Professor Peter Skehan
sheilabj99@yahoo.com
Task-Based Language Teaching 2005
2. The presentation structure…
the rationale of the research
defining the problem
tasks and assessment
Previous findings: weaknesses
research questions and research methods
studies
findings and future plans
implications
3. Research Rationale: Defining the problem Identification of valid, user-friendly sequencing criteria for tasks and test tasks is a pressing but old problem
Grading task difficulty and sequencing tasks both appear to be arbitrary processes not based on empirical evidence (Long & Crookes, 1992)
Not much of an effort been made to define task descriptors in operational terms (see Robinson, 1991)
4. Research Rationale: Tasks and Assessment Grading and sequencing issues assume great importance for testing and assessment of communicative performance
“…to elucidate the potential for using task-based performance assessment to generalize about students’ second language abilities” (Brown et al., 2002, p. 1).
The Brown-Norris matrix (1998; 2002; influenced by Skehan (1996) offers one way of characterising test task difficulty, but lacks obvious connection to a Chinese secondary context
5. Previous findings: weaknesses …previous findings (on task difficulty) were of only moderate support for the proposed relationships between the combinations of cognitive factors with particular task types…
(Elder et al., 2002)
6. This research… investigates the development and use of a prototype task difficulty scheme based on current frameworks for assessing task characteristics and difficulty, e.g. Brown et al, and Skehan (1998).
Hypothesis:
There is a systematic relationship between task difficulty and hypothesized task complexity (see also Elder , 2002)
7. Research questions How can language ability in TBLT in mainland
Chinese middle schools best be assessed?
Is the Brown et al. task difficulty framework appropriate to the mainland Chinese school context? If it is not, then what is an alternative framework?
Is it possible to have a task difficulty framework that can be generalized from context to context?
What are the teachers’ perceptions of task difficulty in a Chinese context?
What are the factors that are considered to affect task difficulty in this context?
8. Research methods (1) a quantitative analysis of ratings of the tasks on the modified task difficulty matrix;
(2) a qualitative analysis of verbal self-report data(introspection) and the focus group interviews on the factors that affect task difficulty.
Methodological triangulation was accomplished by using (a) an analytical task difficulty rating scheme (b) a holistic task difficulty vertical line, (c) verbal self-report (introspection), (d) focus group interviews and questionnaires. Location triangulation was achieved by collecting data from test writers, material developers, experienced teachers and (students) from different regions of China and abroad.
9. Participants and tasks in the development and refining of the task difficulty matrix for prototypical tasks in task-based testing, nine groups of 48 Chinese, English, Swedish test writers, experienced teachers and EFL material developers participated in the rating of 86 tasks, interviews and introspection (verbal self-report)
data of six tasks from 800 students in eight regions (randomly chosen from about a population of 500,000) was collected and analysed.
tasks were designed by Chinese and English test writers, EFL material developers and experienced teachers according to the themes in the Chinese English Curriculum (experimental version, 2001).
10. The research stages 1. First stage: April and May 2004
trial of Norris and Brown task difficulty matrix
2. Second stage: developing and refining the matrix:
Oct 2004: trial on the IPO-CFS task difficulty matrix
3. Third stage: refining the matrix on 24 tasks (Nov 2004)
December 2004: data from 800 students on six tasks
Jan 2005: refining the matrix - ratings of 24 tasks
4. A comparison between Brown et al.’s matrix and the modified matrix
5. Introspection from David, Olov and Prof. B (Feb 2005)
6. Finalizing matrix (Mar – July 2005)
11. Research studies 1. First stage: April and May 2004
To Find out the factors that affect task difficulty among three groups of 26 mainland Chinese English teachers by using Norris et al. (1998)’s task difficulty matrix (Appendix 1)
The results of the test of the Norris et al. approaches to task difficulty among three groups of mainland Chinese English teachers show that there is tremendous disagreement between the Chinese teachers’ and Norris et al.’s predicted difficulty level (Table 1). Among fourteen tasks, both sides agree on only three tasks, Planning the weekend, Shopping in supermarket and Radio weather information which are common general topics in the daily life. The other tasks generated disagreement, especially in relation to cognitive skills, because of different assumptions regarding relevant background and cultural knowledge; different interpretations of the requirements made by different tasks; and different interpretations of how more abstract tasks should be handled.
12. Modified Task Difficulty Matrix
13. Code complexity: linguistic complexity; linguistic input
Cognitive complexity: cognitive familiarity; cognitive processing; amount of input
Communicative stress: time; interaction; context
Task conditions: Language proficiency; language abilities; language skills; culture & other
14. 2. Second stage of research: Oct. 2004
2-teacher trial on the IPO-CFS task difficulty scheme and task analysis on 48 tasks (designed by DL and SL) based on the 24 themes in the Chinese National Curriculum (Table 2)
Findings
Most of the ratings show agreement between the two teachers.
Correlation for the means of both teachers : .65
There is a huge gap (above 6 considered as a big gap between the ratings of Task group 1 (SL) and Task group 2 (DL) of the two teachers) between the ratings of some Group 1 tasks and Group 2 tasks. Table 3 is the analysis of the nine pairs of tasks from the task requirements – to see how demanding the input, the processing and the output are in each task:
15. 24 Themes in the Chinese National English Curriculum (2001) Personal information; Family, friends and people around; Personal environments; Daily routines; School life; Interests and hobbies; Emotions; Interpersonal relationships; Plans and intentions; Festivals, holidays and celebrations; Shopping; Food and drink; Health and fitness; Weather; Entertainment and sports; Travel and transport; Language learning; Nature; The world and the environment; Popular science and modern technology; Topical issues; History and geography; Society; Literature and art
16. Third stage: refining the matrix
Nov 2004: Refining the matrix by collecting data from 5 experienced teachers and test writers, Sunny, Peter, DL, SL and Simon on 24 tasks (designed by test writers and experienced teachers. Table 4 & 5)
Results of the ratings on the refined matrix again show their agreement of the easy and difficult tasks (Appendix 2).
17. December 2004: Six tasks (1, 4, 7, 13-easy and 6, 12-difficult.) tested: data from 800 students in eight different cities and provinces.
Pj: difficulty level of test items. difficulty range: (0.3~0.7). above 0.7, difficult; below 0.3, easy.
18. Six tasks
19. Jan 2005: refining the matrix - ratings of 24 tasks from 6 who have 1) interest; 2) at least are with masters degrees, or even better with PhD degrees; 3) five years of teaching experience or is a test developer or an EFL material writer:
SL, Dodie, Lihy, PS, David, Sunny
Results of the ratings on the matrix (both holistic and analytical to validate the matrix) show a range of correlation from .52 to .83 with only one pair of exception: .34.
21. 4. A comparison between Brown et al.’s matrix and the modified matrix Similarities (5):
Primary research question; Similar purposes; similar design of matrix; an example of an assessment alternative; Sources
Differences (10):
Test Objects; Task Themes; Task Focus; +(-)related to curriculum; Task Selection; Definitions/Labels; Characteristics; Layout; Rating System; Raters
22. 5. Introspection from David, Olov and Prof. B:
they gave detailed verbal self-report data which identified a variety of strategies followed in rating the tasks which help refining the matrix.
6. Finalizing the matrix
the finalized task difficulty matrix sequences tasks from three dimensions, Input, Processing and Output and the following components:
25. A. Content:
1. Information:
l Immediate vs. remote:
1) Here & now vs. there & then;
2) Abstractness vs. concreteness;
3) Familiarity vs. unfamiliarity
2. Amount:
l Total amount:
l Organization:
3. Transformation:
(retrieval and transformation in PROCESSING;
operations in OUTPUT)
26. B. Form:
4. Level of syntax
5. Level of vocabulary
C. Modality: Visual/aural Presentation; Reading; Writing; Listening; Speaking; Others
D. Support: Pictures; Clues; Situation; Authenticity; World knowledge; Personal experience; Common sense; Resources; Tools; Others
27. Plans for Future Research 1. To define the notion of task difficulty
2. To validate the task difficulty matrix and
sequence the 24 themes and prototypical
tasks in the Chinese National English
Curriculum by collecting more data from
raters and students.
3. To define the task descriptors in operational
terms
28. Implications (1) With such a system for estimation of task difficulty, learner performances on carefully sampled tasks can be used to predict future performances on tasks that are constituted by related difficulty components. (Norris et al., 1998:58)
Students with greater levels of underlying ability will be able to successfully complete tasks which come higher on such a scale of difficulty. (Skehan, 1998:184)
29. Implications (2) A fundamental important reason for using pedagogic tasks, sequenced in order of increasing cognitive complexity, as the basis of syllabus design is such a sequencing decision should effectively facilitate L2 development, the acquisition of new L2 knowledge, and restructuring of existing L2 representations. (Robinson, 2001:34)
30. References Brown, J. D., Hudson, T., Norris, J. & Bonk, W. J. (2002). An investigation of second language task-based performance assessments. Second Language Teaching & Curriculum Center, University of Hawai’i at Manoa.
Elder C., Iwashita N., & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: What does the test-taker have to offer? Language Testing, 19,4, 343-368.
Long, M., & Crookes, G. (1992). Three approaches to task-based syllabus design. TESOL Quarterly. 26, 27-56.
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-based second language performance assessment. Language Testing, 19(4), 395-418.
Robinson, P. (2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22 (1), 27 – 57.
Skehan, P. (1996). A framework for the implementation of task-based instruction. Applied Linguistics, 17 (1), 38-62.
Skehan, P (1998). A Cognitive approach to language learning. Oxford: Oxford University Press.