1 / 29

How Multiple choice items distort test takers results in tests of structure in Thailand

How Multiple choice items distort test takers results in tests of structure in Thailand A presentation by Mick Currie and Nanta Chiramanee Prince of Songkla University, Hatyai 14 th December 2007. 1. Overview. The context of the study: Thailand Asia and the EFL/ESL world

kyle-frye
Download Presentation

How Multiple choice items distort test takers results in tests of structure in Thailand

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Multiple choice items distort test takers results in tests of structure in Thailand A presentation by Mick Currie and Nanta Chiramanee Prince of Songkla University, Hatyai 14th December 2007 1

  2. Overview The context of the study: Thailand Asia and the EFL/ESL world Previous studies Our study: Subjects Methodology Findings Implications Your questions and comments How Multiple choice items distort test takers results in tests of structure in Thailand 2

  3. Thailand has not been able to develop widespread communicative skills in English among its population. • Studies have repeatedly found that many teachers do not teach English as a communicative skill. • e.g. Musigrunsi, (2002), • Prapaisit, (2003) • Thongsri, (2005) All identified tests as one of the reasons How Multiple choice items distort test takers results in tests of structure in Thailand 3

  4. What is being tested? Communicative skills or grammar? And how is it being tested? Currie (2007) 97% of students interviewed had been tested in grammar Less than 60% had been tested in writing or speaking Testsoverwhelmingly used multiple choice format Upshur and Palmer (cited in Canale and Swain, 1980) The measurement of linguistic accuracy in Thai students is not an accurate predictor of their ability to communicate Knox (1996) “If the ability to communicate in English is to be taught (and more importantly to be learned) it is vital that this ability also be tested.” How Multiple choice items distort test takers results in tests of structure in Thailand 4

  5. Thailand is not alone. Korea: Li (1998) Japan: Gorsuch (2000) - Multiple choice university entrance examinations affect the way teachers teach and student want to learn China: Liu (2007) -Tests in China concentrate on ‘linguistic competence’ -Multiple choice is the main method in high stakes tests taken annually by more than 8,000,000 students How Multiple choice items distort test takers results in tests of structure in Thailand 5

  6. Little research conducted into whether multiple choice tests effectively assess linguistic knowledge • No published studies comparing stem equivalent items in tests of structure • Rodriguez (2003): a meta analysis of studies into construct equivalence. Found high correlations in stem equivalent items. Identified differences in effects in different domains. • Pike (1979) compared constructed response and multiple choice formats by correlations of reliability (not stem equivalent) • Shohamy (1984) compared stem equivalent items for reading • Cheng (2004) compared stem equivalent items in listening Both found large format effects induced by multiple choice items How Multiple choice items distort test takers results in tests of structure in Thailand 6

  7. Our research methodology Subjects: 152, 1st and 3rd year students from Prince of Songkla University Instruments: A short answer test with 40 structure items 3 multiple choice tests in 3, 4 and 5 option format A post test questionnaire why did subject change their answer on one selected item? Procedure: All subjects sat short answer test Groups of 52, 55 and 45 sat 3, 4 and 5-option multiple choice tests 5/6 weeks later Post test questionnaire after 2nd test How Multiple choice items distort test takers results in tests of structure in Thailand 7

  8. Our research methodology • Example of item construction method. Structure section item # 10 • Constructed response (short answer) item: Stem only • Man: ———— a bank in the university? Student: Yes, it’s opposite the science faculty. • Multiple choice, 3, 4 and 5 option items: Stem and options • Man: ———— a bank in the university? • Student: Yes, it’s opposite the science faculty. • Numbers of subjects • who chose the option in • 3-option 4-option 5-option in the short answer test • a. Is a. Is a. Is (19) • b. Where is b. Where b. Is there (18)* • c. Is there c. Where is c. Where is (39) • d. Is there d. Where (17) • e. Have (17) • * Expected response How Multiple choice items distort test takers results in tests of structure in Thailand 7.1

  9. Our research methodology Analysis: Comparison of subjects scores between the two tests. Comparison of item performance Direct comparison of subjects responses (item by item) in the two tests Controls Control group Control items (4-option) in all m/choice tests Criterion referenced test data 1st year subjects: O-net scores 3rd year subjects: Mid term test Established: Groups of equal ability and no practice effect How Multiple choice items distort test takers results in tests of structure in Thailand 8

  10. Our Findings: Comparison of multiple choice test scores with O-net scores O-net Study Correlation t value 44.69% Control items 44.00% 0.493** 0.538 44.69% Composite m/c 42.80% 0.710** 2.083* *significant at p< 0.05, **significant at p< 0.001 (df=156) (df=157) How Multiple choice items distort test takers results in tests of structure in Thailand 9

  11. Our Findings: *significant at p < 0.001 How Multiple choice items distort test takers results in tests of structure in Thailand 10

  12. Individual scores: 1st & 2nd test 10.1

  13. Individual scores: 1st test vs 2nd test 10.2

  14. Our Findings: How Multiple choice items distort test takers results in tests of structure in Thailand 11

  15. Our Findings: How Multiple choice items distort test takers results in tests of structure in Thailand 12

  16. Our Findings: Correlations 3/4 option: 0.950*** 3/5 option: 0.939*** 4/5 option: 0.995*** df = 9 ANOVA’s C: f = 13.078*** J: f = 5.757** df = 2&149 **significant at p<0. 01 ***significant at p<0.001 How Multiple choice items distort test takers results in tests of structure in Thailand 13

  17. Our Findings: **significant at p < 0.001; df = 9 How Multiple choice items distort test takers results in tests of structure in Thailand 14

  18. Our Findings: • High ability subjects: • Maintained more correct and less incorrect answers between tests • Selected the correct response when their answer from the 1st test was not among the options in the multiple choice test, 3 times out of 4 • High and middle ability subjects: • Were twice as likely to switch from their incorrect answer in the first test to the correct option, than low ability subjects • Low ability subjects: • Selected an incorrect response when their answer from the 1st test was not among the options in the multiple choice test, 3 times out of 4 How Multiple choice items distort test takers results in tests of structure in Thailand 15

  19. Our Findings: • Subjects in the 3-option group • Were more successful at selecting the correct response when their answer from the first test was not among the options than subjects in the 4 and 5-option groups (pattern J) • Switched between incorrect options when their original response was among the options, half as often as did subjects in the 4 and 5-option groups (Pattern C) But: Overall the number of options had very little effect How Multiple choice items distort test takers results in tests of structure in Thailand 16

  20. Our Findings: • Why did the subjects change their answers between the two tests? • Knowledge • Learning • Cued recall • Test taking strategy/technique • Blind guessing How Multiple choice items distort test takers results in tests of structure in Thailand 17

  21. Our Findings: First test Second test How Multiple choice items distort test takers results in tests of structure in Thailand 18

  22. What are we to make of these results? • Our Conclusions: • The multiple choice format enabled the subjects to achieve higher scores than the short answer test How Multiple choice items distort test takers results in tests of structure in Thailand 19

  23. What are we to make of these results? • Our Conclusions: • The multiple choice format enabled the subjects to achieve higher scores than the short answer test • The improvements were only weakly correlated with the first test suggesting that language ability was not responsible for the improvement How Multiple choice items distort test takers results in tests of structure in Thailand 20

  24. What are we to make of these results? • Our Conclusions: • The multiple choice format enabled the subjects to achieve higher scores than the short answer test • The improvements were only weakly correlated with the first test suggesting that language ability was not responsible for the improvement • The high correlations between the two tests How Multiple choice items distort test takers results in tests of structure in Thailand 21

  25. What are we to make of these results? • Our Conclusions: • The multiple choice format enabled the subjects to achieve higher scores than the short answer test • The improvements were only weakly correlated with the first test suggesting that language ability was not responsible for the improvement • The high correlations between the two tests are misleading How Multiple choice items distort test takers results in tests of structure in Thailand 21.1

  26. How Multiple choice items distort test takers results in tests of structure in Thailand 22

  27. What are we to make of these results? • Our Conclusions: • The multiple choice format enabled the subjects to achieve higher scores than the short answer test • The improvements were only weakly correlated with the first test suggesting that language ability was not responsible for the improvement • The high correlations between the two tests are misleading • Only around 27% of the answers from the 1st test were chosen from the multiple choice options in the 2nd test • 73% of the 2nd test option selection was forced or induced by the test format How Multiple choice items distort test takers results in tests of structure in Thailand 23

  28. What are we to make of these results? • Our Conclusions: • The multiple choice tests grossly distorted the measurement of the test takers performance • What was actually being measured was largely the subjects’ ability to deal appropriately with the multiple choice format • Based on this study, the multiple choice format should not be used in tests of language structure How Multiple choice items distort test takers results in tests of structure in Thailand 24

  29. How Multiple choice items distort test takers results in tests of structure in Thailand Your questions and comments are welcome. END

More Related