How China Secures its Tests

1. How China Secures its Tests Breakout Session Alex TONG, ATA

2. How Serious is ‘Test Cheating’ in China • Common Fraud in testing administration in China • Underlying reasons for the popularity of cheating and administration ‘fraud’ • What actions has China authorities taken for the last few years • Ministry of Justice Licensure exam - a case study

3. How serious is cheating in China? • An online survey in June 2008 conducted to graduates on exam-cheating in Universities has alarming results: • Have you cheated in your exams? Yes – 51%; With pride – 11% No – 49%; Would Stop others – 11% • Agrees to ‘Cheating rather than Failing’: Agree – 74%; Disagree – 9%; http://www.zhijizhibi.com/op_FullResultReport!questionnaireid_161959039!charttype_2.html

4. Common ‘Fraud’ in Test administrations in China

5. Common fraud found in Test Administrations • Impersonations • Leakage of exam items and forms • Onsite organized cheating/assistance • Test results integrity

6. Common fraud found in Test Administrations • Impersonation • Organized proxy services, • Fake ID easily accessible, • Administrator not trained to verify fake documentations • Insignificant legal consequences

7. Common fraud found in Test Administrations – Cont. • Items/forms leakage • Brain-dump sites acquiring/selling items/forms • Candidates upload test items they have taken to share in forums • System fraud • Insecure item formats (unified test with single form, Paper based, etc…) • Too many human interventions in items/forms processing

8. Common fraud found in Test Administrations – Cont. • Onsite cheating/assistance behavior • ‘Consulting’ neighboring seats • ‘Bringing’ in reference materials • Communications for external assistance

10. Common fraud found in Test Administrations – Cont. • Test Results Security • Scorer fraud – altering individual scores • Local results – insecure storage media: • paper score sheet, • removable storage devices • Database being hacked to alter test results • In 2008 Ministry of Health identified and uncovered a group of 200+ individuals assisted in altering the exam result databases

11. Underlying reasons for the popularity of cheating and administration ‘fraud’

12. Underlying Reasons for the Popularity of Cheating and Administration Fraud • Fierce competition in job market • High dependency of qualifications • Too many human interventions in test administration process • Cultural reasons and lacking of legal consequences

13. Underlying Reasons for the Popularity of Cheating and Administration Fraud • Fierce competition in job market • Resulting from the reforms in China education system, there is an increasing number of university graduates in China year on year: 5.7 mil (2008) to 6.2 mil in 2009 • According to the Ministry of Labor there are 24 million people seeking for jobs in 2009 while there will be 12 million vacancies

14. Underlying Reasons for the Popularity of Cheating and Administration Fraud • High dependency of qualifications • Large number of job applicants for any job; • Tradition of certification as an entrance ticket for anything • Test administration process • Unified test administration using single form • Too many processes involving human interventions

15. Underlying Reasons for the Popularity of Cheating and Administration Fraud • Cultural reasons and lacking of legal consequences • ‘Loosing Face’ if failed, cheating as an insurance; • ‘helping each other’ is a traditional Chinese virtue • ‘Cost of Failing’ too high for existing practitioners • Maintaining ‘GuanXi’ (relationship) through providing ‘special assistance’ i.e. fraud in administrations • Insignificant legal consequences until late 2008

16. Actions taken by Chinese authorities in the last few years

17. Actions Toward Secured Testing • Legislation – Testing Act being drafted in 2008 • Standardize testing procedures • Disciplinary actions on cheating behavior (max. of 7 years of imprisonment) • Covers all aspects of testing, from test publishing, item security, test proctoring, test result evaluation, through remediation for the penalized • Create and maintain large item banks • The National Item Bank of Ministry of Education • The National Item Bank of Ministry of Personnel • The National Item Bank of MOF and MOE http://news.xinhuanet.com/legal/2008-03/21/content_7832471.htm

18. Actions Toward Secured Testing • Leveraging technology-based measures for supervision, authentication and monitoring of test administration processes • Minimize human factors and exposure to test items • Converting paper-based exams into computer-based test administration

19. Technology-based Test Administration • Online-offline-online test mode • Secured Data Transfer • Secured Content Display • On-site demographic Verification

20. Security built throughout the process Pre Test Testing Post Test Secured Test Delivery System 128-bit symmetric encryption deployed throughout operation processes from pre-test, test delivery and post-test stages Off-line Online Online

21. Authorization Form/ Items Pre Test Testing Post Test 数 据 信 息 报 名 信 息 时 间 信 息 Secured Data Transfer Test Taker Password Test Station Test Admin. Station Exam Starts Shredder

22. Secured Content Display Randomized item display Randomized sequence of choices Performance based items Single item displayed on screen at a time Pre Test Testing Post Test

23. Onsite Demographics Creation 登录 Pre Test Testing Post Test On site photo-identity verification Candidates arriving Onsite Photo taking Photo- demographics

24. Test development/production Processes Personnel involved in test development/production sign legal binding Non-Disclosure Agreement which held them with legal responsibilities to protect information they are working on; Separation of teams who work with test contents, no individuals would have access to and/or able to associate their works on hand with any particular test titles or the entire item bank Role-based Security Control (Access Level) Minimize Human Factors and Exposure to Test Items

25. Minimize Human Factors and Exposure to Test Items Form Assembly Randomized items packaging from item bank based on blueprint of exam Standardized test item types are stored in centralized application servers, and will only be downloaded dynamically before actual test time. Items being downloaded from the application server will compose the test form ‘live’ according to the ‘blue-print’ of the exam provided on the server

26. Conversion of PPT to CBT China Securities & Regulatory Commission once-a-year paper based test turned to CBT in 2001. China Future’s practitioner Test converted to CBT in 2004 Ministry of Education revolutionized its National IT Level Exam from paper based into simulation-based performance test (2006) China Banker’s Test opted for CBT in 2006. College English Tests (11 million tests/year) to be converted to CBT in 2009 And more to come…

27. Ministry of Justice – A Case Study Ms Xia Lu, Chief Exam office, Ministry of Justice

28. MOJ Licensure Exam - A Case Study • Background of MOJ exams • Common fraud found in MOJ exams • Actions being implemented in MOJ exams today • What’s next?

29. Background of MOJ Exams • Launched in 2002 for China lawyers, judges and prosecutors qualifications • Administered across the country once a year with a unified A/B form • Extremely high stake qualification leading to prestige social status • Currently 300K examinees each year • 4 exams all administered through paper forms • Total of 7 hours of test conducted in one day

30. Common fraud found in MOJ exams • Handled over 1,000 verified fraud and complaints in the exam administration since 2004 • Reference Materials/ ext. communication 42.6% • On-site Communications during exam 36.2% • Identical set of wrong-answers (copying) 14.5% • Impersonation 5.3% • Others 2.3%

31. Actions Taken • Create and fine tune Regulations and Guidelines for legal actions against exam fraud (Guidelines and Actions for Exam Indiscipline behaviors, 2002) • Aggravate disciplinary actions against exam fraud • Leverage technologies to monitor, avoid and prevent misbehaviors in exam administrations • Communicate and Educate public of the importance of a fair exam and the consequences of exam fraud

32. A Legal Case against Exam Fraud • In Sep 2007, a group of 22 test takers were caught and admitted to buying a set of live 2007 MOJ exam questions • A MOJ officers were put behind bars for leaking the exam questions to his old friend who has a daughter who failed MOJ exam twice • The boyfriend of the daughter was also put behind bars for selling the items to 22 examinees at price of $100K each

33. References • http://www.longhoo.net/gb/longhoo/news/edu/userobject1ai625394.html Two Ministries invested 100million in building test item banks to ensure security in exams • http://learning.sohu.com/20080123/n254827913.shtml 《Exam Law》 Would this Stop cheating in Exam • http://internal.northeast.cn/system/2007/04/09/050767455.shtml Ministry of Education: National Exam item bank launched, pushing for the legislation of Exam Law • http://dyh.czkj.gov.cn/Article/ShowArticle.asp?ArticleID=213 University Entrance Exam: To build large national item bank to replace ad hoc items/forms building • http://cks.mof.gov.cn/03zhengwen.asp?news_id=3955 Ministry of Education built National Item bank - Creating fairness to the exam takers • http://news.xinhuanet.com/legal/2008-03/21/content_7832471.htm Ministry of Justice put three ‘cheaters’ behind bars

34. Thank You !