Designing and Developing Useful Language Tests

1. Designing and Developing Useful Language Tests Lyle F. Bachman Department of Applied Linguistics & TESL University of California, Los Angeles

2. Uses of Language Tests • to make inferences about individuals’ language ability • to make predictions about individuals’ ability to use language in contexts outside the test itself • to make decisions about individuals

3. Correspondences between Language Test Performance and Language Use

4. Characteristics of the Language Use Task Characteristics of the Language Test Task Language Use Language Test Performance Characteristics of the Test Taker Characteristics of the Language User

5. Characteristics of the Language Use Task Characteristics of the Language Test Task Language Use Language Test Performance Language Ability of the Test Taker Language Ability of the Language User

6. Characteristics of the Language Use Task Characteristics of the Language Test Task Language Use Language Test Performance Language Ability of the Test Taker Language Ability of the Language User

7. A Framework of Task Characteristics Language Use Task an activity that involves individuals in using language for the purpose of achieving a particular goal in a particular situation (Bachman & Palmer, 1996, p. 44)

8. A Framework of Task Characteristics Target Language Use (TLU) Domain a set of specific language use tasks that the test taker is likely to encounter outside of the test itself, and to which we want our inferences about language ability to extrapolate (Bachman & Palmer, 1996, p. 44)

9. A Framework of Task Characteristics Task Characteristics • Setting: the physical circumstances under which either language use or language testing takes place. Includes the physical characteristics of the setting, the participants involved, and the time of the task.

10. A Framework of Task Characteristics Task Characteristics • Rubric: the context for the task; includes those characteristics that provide the structure for the task, and constrain how language users or test takers are expected to respond to these tasks

11. A Framework of Task Characteristics Task Characteristics • Input: material contained in the task, which test takers need to process in some way, and to which they are expected to respond.

12. A Framework of Task Characteristics Task Characteristics • Expected response: the language use that is expected, given the way in which the rubric, or context, for the task is configured, and the particular input that is provided

13. A Framework of Task Characteristics Task Characteristics • Relationship between input and response: • Reactivity: the degree of reciprocity, or interaction involved, • Scope: the amount and range of input that needs to be processed in order to respond, and • Directness: the extent to which the response can be made by using information in the input by itself, or whether the language user or test taker must also rely on information in the context or in his or her own real world knowledge.

14. Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe these tasks in terms of their task characteristics.

15. Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe TLU tasks in terms of their task characteristics.

16. Task Characteristics Framework TLU Task Type A (letter) TLU Task Type B (memo) TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 3 responding to customer complaints on the phone Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing

17. Developing Test Tasks • Modify TLU task types for use as test tasks • Create original test tasks

18. TLU Task Type A (letter) TLU Task Type B (memo) Modify TLU Task Type to Develop Test Task Specifications Test Task Specifications A (write formal letter to customer) Test Task Specifications B (write memo to co-worker)

19. Developing Test Tasks • Modify TLU task types for use as test tasks • Create original test tasks

20. Distinctive Characteristics of TLU Task Type Create Specifications for Original Test Tasks Test Task Specifications

21. Example Test Development Project Project setting: telephone company needs to hire new employees

22. Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe these tasks in terms of their task characteristics.

23. Task 1 writing promotional material Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 3 responding to customer complaints on the phone Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing Example TLU Domain PR Department of a Phone Company

24. Selecting TLU Tasks for Use in Designing Test Tasks • Some TLU tasks may be essential to performing the job

25. Task 4 responding to customer complaints in writing Task 6 giving complaint information to co-workers in writing Example TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 2 phone discussions with clients, co-workers Task 3 responding to customer complaints on the phone Task 5 giving complaint information to co-workers by phone

26. Selecting TLU Tasks for Use in Designing Test Tasks • Some TLU tasks may not require the areas of language ability we want to measure. • Some TLU tasks may not be appropriate for all test takers.

27. Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe these tasks in terms of their task characteristics.

28. Task Characteristics Framework TLU Task Type A (letter) TLU Task Type B (memo) TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 3 responding to customer complaints on the phone Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing

29. Developing Test Tasks • Modify TLU task types for use as test tasks. • Create original test tasks.

30. Developing Test Tasks • Modify TLU task types for use as test tasks • Create original test tasks

31. TLU Task Type A (letter) TLU Task Type B (memo) Modify TLU Task Type to Develop Test Task Specifications Test Task Specifications A (write formal letter to customer) Test Task Specifications B (write memo to co-worker)

32. Developing Test Tasks • Modify TLU task types for use as test tasks • Create original test tasks

33. Distinctive Characteristics of TLU Task Type Create Specifications for Original Test Tasks Test Task Specifications

34. Conclusion Bachman-Palmer Approach to Language Test Development • Provides test developers with a • theoretically grounded and principled • basis for developing and using • language tests.

35. Conclusion Bachman-Palmer Approach to Language Test Development • Provides test developers with an • understanding that will enable them • to make their own decisions about • either selecting or developing a • language test that will be useful for • their particular situation.

36. Conclusion Bachman-Palmer Approach to Language Test Development • Requires test developers to consider the • fundamental correspondence between • language test performance and non-test • language use.