STRATEGIES TO REDUCE TEST FRAUD

1. STRATEGIES TO REDUCE TEST FRAUD Preventing, detecting, and investigating test security irregularities Gerard Seinhorst Head, Language Office Defence Intelligence and Security Institute

2. INTRODUCTION

3. PRESENTATION OVERVIEW • Test Fraud – What is it and why is it a concern? • Cheating – What do we know about it? • Strategies to minimize test fraud • Prevention • Detection • Follow-Up Actions

4. TEST FRAUD • 1. Cheating: • “any action taken before, during, or after the administration of a test or assignment, that is intended to gain an unfair advantage or produce inaccurate results” (Cizek, 2012) • 2. Test Piracy: • stealing, copying or memorizing test forms, items, prompts, or other secure testing materials, usually with the intention to make these available to future test takers.

5. TEST FRAUD – Why is it a concern? • Threat to validity Construct-irrelevant variance • Reputation of testing programme • Fiscal implications • redevelopment of testing materials • test retake sessions • investments in test security • Negative impact on validity of scores obtained by other candidates • Denying opportunities to other candidates

6. CHEATING – When? • Cheating tends to occur when: • there is a good opportunity (cheating is easy to do) • there is a small likelihood of getting caught • if caught, the penalties are light • faculty appear to take a casual or lenient attitude towards cheating • the stakes are high A thing worth having is a thing worth cheating for W.C. Fields

7. CHEATING – How? • Variety of forms

8. CHEATING – How?

9. CHEATING – Resources Cheating Sites: http://www.schoolsucks.com/ http://www.academon.com/?cf=uk http://www.writework.com/ http://www.academicintegrity.com/

10. CHEATING – How much? • Cheating is on the rise • Research studies: • On average, between 3-5% oftest takers engage in cheating on any occasion • 100% of 14 educational organizations participating in a survey found firm evidence of any teachers or school administrators cheating on behalf of their students

11. WHEN TEACHERS ARE TEST ADMINISTRATORS • Overtly and covertly giving help to test takers: • changing responses on answer sheets after testing • leaking test questions before testing • applying non-standard testing conditions • cueing students on incorrect answers • giving students extra time on tests • filling in answers left blank by students • suspending low-ability students on testing days • giving inappropriate instruction:“teaching to the test”

12. CHEATING – Why? • Top-5 reasons why test candidates cheat: • competitiveness: pressure to perform / grade pressures • poor time management / exam preparation • lack of self confidence: anxiety about test content and test format • “cheating is easier than studying” • cheating culture (thrill / collective cheating)

13. CHEATING BY EDUCATORS • Material rewards (bribes) • Pressure to promote strong performance by their students • Indifference • Justifiable response to standardized tests • Compassion for their students • Ignorance

14. TEST FRAUD PREVENTION If a man defrauds you one time, he is a rascal; if he does it twice, you are a fool. Author unknown

15. PREVENTING TEST FRAUD Focus on prevention, rather than remediation Acknowledge that cheating is going to occur and is problematic

16. 6 STRATEGIES TO REDUCE TEST FRAUD • Develop faculty and student integrity • Develop and implement a Security Plan • Ensure that administration staff are properly trained • Protect testing materials against piracy • Administer tests in controlled environments • Maximize probability for detection

17. 1. Develop Faculty & Student Integrity • Create culture where it is inappropriate to engage in any form of cheating or piracy • Respond to cheating when it does occur • Refrain from inappropriate test preparation activities • Eliminate test anxiety: Familiarize candidates with • Test format, length, etc. • Scoring criteria • Test admin procedures • Re-test policy • Appeal procedure • Examination honour code / non-disclosure agreement

18. Examination Honour Code Please read the following and provide your signature: I agree to answer the questions on this assessment without using aids that are not permitted and without obtaining assistance from another person or via electronic means. I agree to not share my answers or any information about the assessment content with anyone during or after the assessment. I accept that if any of these conditions are violated the result will be no credit (0 points) for this test. Signature

19. 2. Develop & Implement Security Plan • Roles and responsibilities • Secure management of testing materials • Prevention • Test administration procedures • (Im)permissible behaviour • Detection and investigation of irregularities • Sanctions for misconduct

20. 3. Ensure admin staff are properly trained • Lack of training results in distrust of testing and a misunderstanding of the reliability and validity of standardized testing • Training should include: • common cheating methods • test administration protocols • how to act in case of cheating • impact of test security irregularities • role modelling

21. 4. Protect testing materials against piracy • Safeguard all secure materials • Limit exposure of test items • Use appropriate (not easily-corruptible) test construction and delivery methods

22. a. Safeguarding testing materials • Account for all secure materials before, during, and after testing • Store all testing materials (including draft items and test results) in multiple, locked cabinets • Password protect all electronic files • Number test booklets AND answer sheets • Shred all obsolete testing materials • Limit access • Pre-package test booklets (shrink-wrapped or in sealed envelopes) • Maintain a clean-desk policy

23. Clean-desk policy…

24. b. Limit exposure of test items • Reduce amount of testing • Narrow testing windows • Use screening tests • Use unique make-up tests • Use an item bank • Periodically introduce new test forms • Use computer-adaptive tests

25. c. Test construction & delivery strategies • Design tests with security in mind: • Include as many items/prompts as feasible • Use more constructed response items • Develop multiple versions and forms ( validity issue) • Randomize answer choices (in MC testing) • Use as many plausible answer choices as feasible • Use computer-based / computer-adaptive testing • Possible tradeoffs in terms of psychometrical disadvantages, lower efficiency and added costs

26. 5. Administer tests in controlled environments • Keep high-stakes testing separated from teaching • Avoid that teachers test their own students Establishing a controlled environment…

27. Standardized administration procedures • Seating plan

28. Randomly assign seats Beware of the “flying V” answer copying formation (Cizek, 1999)

29. Standardized administration procedures • Seating plan • Monitoring

30. Monitoring Monitoring is still the most effective way to prevent and detect misconduct during testing

31. Standardized administration procedures • Seating plan • Monitoring • Prohibited behaviour / items / aids • talking, walking, gesturing, electronic devices, reference materials, crib sheets, hats, sunglasses, bags • Account for all secure materials (incl. scrap paper and blank answer sheets)

32. Special considerations for CBT / CAT • Security risks: • Inappropriate proctor assistance • Test piracy (hacking) • Lack of familiarity • Absence of secure browser • Technology dependent

33. 6. Maximize probability for detection • Begin monitoring for irregularities prior to test administration • Monitor the Internet for pirated test items • Qualified proctors should monitor for irregularities during the test administration • Double marking to prevent educator • cheating after testing • Routinely conduct data forensics: • Ratio analysis/erasure analysis • Item-response pattern analysis • Test-score analysis • See Van der Linden (2011) for a list of psychometric techniques

34. FOLLOW-UP ACTIONS Five Fs of follow-up: fast, firm, fair, frequent, famous

35. Response and Investigation • Security Incident Response Plan • Establish a standard or trigger for an investigation • Use appropriate sanctions • formal reprimand • re-taking the test • informing stakeholders • withdrawal of certificate/diploma • legal action

36. CONCLUSION • Teacher shows ultimate cheating technique…

37. THANK YOU FOR LISTENING Proctoring – not an easy job…

38. REFERENCES / FURTHER READING Cizek, G.J. (1999). Cheating on Tests: How to Do it, Detect it, and Prevent it. Lawrence Erlbaum Associates: Mahwah, New Jersey. Cizek, G.J. (2012). Ensuring Integrity in Testing: Context, Responsibilities, and Recommendations. Retrieved August 21, 2013 from http://www.gomiem.org/ files/handouts/keynote_thursday-cizek_cheating_2012-02-23.pdf. Impara, J.C., Kingsbury, G., Maynes, D., and Fitzgerald, C. (2005). Detecting Cheating in Computer Adaptive Tests Using Data Forensics. Retrieved August 21, 2013 from http://www.caveon.com/articles/NCME-05.pdf Noah, H.J. and Eckstein, M.A. (2001). Fraud and Education: The Worm in the Apple. Maryland, USA: Rowman and Littlefield. Van der Linden, W.J. and Sotaridona, L. (2004). A Statistical Test for Detecting Answer Copying on Multiple-Choice Tests. Journal of Educational Measurement, Vol. 41, No. 4, pp. 361–377. Whitley, B.E. and Keith-Spiegel, P. (2002). Academic Dishonesty: an Educator’s Guide. Lawrence Erlbaum Associates: Mahwah, New Jersey. Wollack, J.A. & Fremer, J.J. (Eds.). (2013). Handbook of test security. New York City, NY: Routledge.