Describe and evaluate the use of multiple choice question for testing listening comprehension

1. Describe and evaluate the use of multiple choice question for testing listening comprehension Presenter: Luong Thi Phuong Nhi (MA) Faculty of English for Specific Purposes Foreign Trade University

2. CONTENT Introduction I Discussion II Evaluation III Conclusion IV

3. Introduction According to McNamara (2000), language tests play a significant role in various aspects of many people’s lives. A great deal of research on language assessment have been undertaken in language literature. The appearance or the growth of a language testing method has resulted from a certain language teaching and learning approach. Multiple choice tests have been still perceived as a discrete-point testing method, resulting from the model of discrete-point approach. This presentation first discusses the communicativeness of the multiple choice item and then evaluate the use of multiple choice item for testing listening comprehension in terms of its practicality, reliability, validity and backwash.

4. Description of Multiple choice items There have been many forms of multiple choice items; however, their fundamental structure is described as the following (Brown, 2004; Hughes, 2003): There is a stem which presents a stimulus: Indoor heating systems have made ______________ for people to live and work comfortably in temperate climates. and a number of options or alternatives(normally ranging from three to five) – one of which is akey (a correct response), the others being distractors: A. it is possible B. possible C. it possible D. possibly (The above example cited from TOEFL success, 1996: 424)

5. Discussion McNamara (2000) states that multiple-choice format is regarded the most important among several types of fixed response format. The use of multiple choice format for testing listening was widely proposed during the period when the discrete-point approach gained its favor. The requirement for new or innovative language test methods appears when the concept of listening comprehension changes. Under the new notion of listening comprehension, multiple choice items adopt “new features”. Although the test format remains almost the same, the construct is quite different. It is therefore a misconception if someone still refers multiple choice tests as “discrete-point tests” (Buck, 2001).

6. The “communicativeness” of multiple-choice test The idea of communicative teaching actually leads to a new shift in language testing approach. The language assessment focuses on the measurement of “how much a person knows about the language” and how he/she can “use it to communicate effectively” in the target language use context (Buck, 2011; Weir, 1990). Such features as the real-world context of the language use, the authenticity of the tasks or the texts become a matter of concern in communicative testing (Buck, 2001). It could be said that the demands of the tests and their use of real-world context as well as of more authentic task or texts make multiple choice tests “communicative” in character.

7. Demand of the multiple-choice test Although the test-takers are only asked to put a tick against the questions they choose in multiple choice test, there is a need of interaction between the test takers and the task. A variety of listening sub-skills may be assessed in multiple choice tests (Buck, 2001). The test of listening sub-skills can range from “understanding at the most explicit literal level, making pragmatic inferences, understanding implicit meanings to summarizing or synthesizing extensive sections of tests” Each kind of listening sub-skills probably places a certain sort of demand on the test-takers. In other words, the multiple choice tests would demand “some meta-cognitive processing skills” associated with the test method (Brown, 2004).

8. Context There has been much criticism on the use of multiple choice tests in “isolated” contexts or “de-contextualized” situations. The use of multiple choice items in listening tests, however, might depend on the purpose of listening assessment. Given the tests being used for listening comprehension, multiple choice tests need provide contexts of the language use in real-life situations. In some standardized tests, say TOEFL or TOEIC, multiple choice items are used to test not only phonology or paraphrase recognition but also responsive and extensive listening. Both responsive and extensive listening require the context of language use essential for the listeners to comprehend the whole texts and perform the required tasks.

9. Authenticity In reality, a communicative test requires the test takers to perform a real-world task in a certain real-life communicative context. Multiple choice test could be perceived as a “communicative” one because it seems to include both authentic tasks and texts. Authentic tasks: Listening to monologues, lectures or brief conversations are likely to be common tasks in multiple choice tests. The test-takers are normally required to answer to a set of comprehension questions after their listening (Brown, 2004). Authentic texts: To create the authenticity of the test texts such natural speech features as assimilation and elision as well as hesitation phenomena could be found in the multiple choice listening tests (Hughes, 2003). The genuineness of the texts and the authenticity of tasks seem to be the main focus of TOEFL or TOEIC tests, which still use the multiple choice format to assess listening skills.

10. Evaluation The question of how to make a good or effective tests attract attention of both test constructors and teachers. There are some major criteria for “assessing” a test, for example, practicality, reliability, validity and backwash. Those criteria are normally “evaluated” separately on the process of “testing a test” (Brown, 2004).

11. Practicality Making or scoring in multiple choice item tests are “rapid” and “economical” (Cohen, 1998; Hughes, 2003). The tests provide “predetermined correct responses” and “time-saving scoring procedures” offering the raters an “easy and consistent process of scoring and grading” (Brown, 2004). However, regarding preparation phrase, the practicality of multiple choice test is likely to be in question since it takes more time, money and effort to prepare multiple choice questions than open-ended items. Although multiple choice tests seem to be simple to design, they are of great difficulty to construct correctly due to their complexity (Brown, 2004; Buck, 2001; Cohen, 1994). There is a common view that the construction of multiple choice items requires trained, skilled or experienced test designers and all items need trailing or pre-testing before being used in a test, especially high-stake assessment (Alderson, 2001; Brown, 2004; Buck, 2001; Weir, 1993). In high-stake assessment, it is probably efficient to administer and score if using multiple choice items (Brown, 2004; McNamara, 2000).

12. Reliability In multiple choice tests, scoring could be perfectly reliable (Cohen, 1994; Hughes, 2003). Additionally, in multiple choice tests, the issue of scorer’s subjective assessment is out of concern. The scorers are not permitted to give any judgment when marking candidates’ answers (Weir, 1998). However, the test takers scores gained in multiple choice tests may be of concern. When doing multiple choice tests, the candidates can get some or most of the correct answers by using some test-taking strategies like eliminating “implausible choice” or just simply by guessing (Alderson, 2000; Brown, 2004; Cohen, 1994; Hughes, 2003; Weir 1988, 1990). The type of test and the level of reliability also depend on the number of options in each item. To keep multiple choice items more reliable, it is suggested that three or four options or alternatives should be presented (Harrision, 1983; Hughes, 2003); candidate should be required to give their reasons for marking their choice (Alderson, 2000). Those two suggestions might make the test more reliable but may affect its practicality.

13. Validity Multiple choice items could serve as tests of receptive skills without requiring the test-takers to show their productive skills: writing or speaking skills (Hughes, 2003). Multiple choice tests thus may bring an incorrect picture of the candidates’ language ability if the assessment of their ability is only based on the test of receptive skills. The concern about the uncertainty of their validity arises when multiple choice items are used to measure language ability. Test-takers’ performance on multiple choice items is likely an “unreal task” since in real-life situations they are not forced to show their comprehension by choosing the best options among those suggested (Weir, 1990). Another concern about the validity of multiple choice tests is that while candidates are asked to complete a listening task, they are required to read and keep in mind four or more alternatives for each listening item (Hughes, 2003; Weir, 1990). It is however hard to decide whether the candidates give the wrong answer because of their incomprehension of the text they listened to or just because of their misunderstanding of the suggested questions or alternatives (Weir, 1988, 1990). The problem with its format creates a sense that the multiple choice test tends to be “an invalid method for assessing comprehension” (Weir, 1990).

14. Backwash The use of multiple choice items for testing listening comprehension may have some negative effect on teaching and learning. There might be a tendency to have some training in “improving educated guesses rather than in learning the language” (Hughes, 1989; Weir, 1990 cited in Cohen 1994). For the purpose of score improvement, much effort would be put on the training test taking techniques rather than learning to improve their listening ability. According to Weir (1993), such improvement does not mirror an increase in learners’ language command, but “an enhanced ability” of doing multiple choice test.

15. Conclusion The usefulness of multiple choice items for language testing in general and testing of listening comprehension in particular has been ongoing debatable. It should be flexible and sensible when using multiple choice items for testing listening. Depending on the type and the purpose of testing, teachers should know how to make the best use of multiple choice tests. Despite some advantages offered by this test tool, the overuse of multiple choice tests in the class may impose some negative effect on teaching and learning.

16. References Alderson, J.C.(2000). Assessing reading. Cambridge: CUP. Bachman, L.F. & Palmer, A. (1996). Language testing in practice. Oxford: OUP. Brown, H.D. (2004). Language assessment: Principles and classroom practices. Person Education, Inc. Buck, G. (2001). Assessing listening. Cambridge: CUP. Cohen, A. (1994). Assessing language ability in the classroom. Boston, MA: Heinle and Heinle. Harrison, A. (1983). A language testing handbook. London: Macmillan. Hughes, A. (2003). Testing for language teachers. Cambridge: CUP. McNamara, T. (2000). Language testing. Oxford: OUP. Rogers, B. (1996). TOEFL success. Peterson’s Princeton, New Jersey. Weir, C.J. (1988). Communicative language testing. The university of Exeter. Weir, C.J. (1990). Communicative language testing. New York. Prentice Hall. Weir, C.J. (1993). Understanding and developing language tests. New York. Prentice Hall.