The need for international guidelines on computer-based testing and the Internet.

1. The need for international guidelines on computer-based testing and the Internet. Dave Bartram Research Director SHL Group plc

2. Why do we need International Guidelines? • Individual countries are no longer ‘closed’ systems. • Use of WWW for distance assessment – especially in Recruitment and 360 Feedback. • Mobility of test users and cross-border access to materials, especially in EU. • Impact of globalisation on assessment procedures • Variations between countries in • standards adopted and training provision • regulation of access to test materials • statutory and non-statutory controls over testing and test use

3. A Thought • It took: • 38 years for radio • 13 years for TV • 4 years for WWW ……….to reach 50m Users. Now it is over 500m

4. Issues • Internet is an international medium. It no longer makes sense to try to control testing with restrictions in one country if there are no restrictions in the next country. • Rapid proliferation of testing on the Internet. • Standard inventories being administered remotely • Timed ability tests being put on the Internet in un-timed and unproctored modes • Lots of DIY tests appearing – some for serious use, many for ‘amusement’ – with little or no information about provenance for the user.

8. Risk Management • Testing is an exercise in risk management. • For example, we balance: • Risks of item exposure against benefits of volume assessment • Risk of collusion on self-report measures against logistic benefits and cost savings of unsupervised administration • Internet changes the balance and creates need to review some of these risks and consider new risk mitigation strategies. • Good practice is about managing risk appropriately

9. Risk is managed by exercising control • Control over prior practice – informing test takers, starting from a level playing field, avoiding uncontrolled exposure of sensitive items. • Control over materials – psychometric quality, security, updating, everyone using the same versions. • Control over test takers – authentication: knowing who is taking the test • Control over test conditions - ensuring conformity to critical administration constraints: e.g. timing, clarity of display etc.

10. What are the functions of assessment supervision? • To authenticate identity of test takers • To establish rapport with the test taker • To ensure the standardised conditions are followed • To ‘validate’ results (prevent cheating and collusion) • To deal with problems • To ensure security of materials

11. What are the functions of assessment supervision? • To authenticate identity of test takers • To establish rapport with the test taker • To ensure the standardised conditions are followed • To ‘validate’ results (prevent cheating and collusion) • To deal with problems • To ensure security of materials These are difficult to manage remotely

12. Internet Test Administration modes Insecure mode Moderately secure mode Secure modes

13. Controlled = Moderate control + Major cost benefits Supervised/ Managed = traditional & comfortable Open = High volume, but low control HIGH STAKES TESTING & ZONES IN THE RECRUITMENT FUNNEL

14. Cognitive ability testing: supervised • Traditional tests need high security to protect items. • Most tests are timed or need good time control • Standard HTML web pages not good for this • Need downloadable aplets which can run even if server link is down. • Difficult to manage adaptive testing unless bandwidth is high: Managed Mode allows for download of item sets.

15. ISSUES OF CHEATING & FAKEABILITY • What is it sensible to put in the Open end of the selection funnel? • Practice tests • Work samples and tools aimed at optimising self-selection in/out • Cognitive tests can be administered in ‘open’ modes but: • There is no control over cheating • There is no candidate authentication • There is no means of safeguarding item security • OK for low stakes assessment with item sets where item exposure is of no concern. • E.g. Self-assessment for career guidance

16. MOVING TIMED TESTING UP THE FUNNEL • Constrained random generation of parallel tests from large item banks • Multiple versions of items • Unique test delivered to each candidate • Standardised scoring through IRT item calibration • Used unsupervised as a short-form pre-sift, with a supervised retest at Assessment Centre for final short-list. • Supported by ‘honesty contract’.

17. ZONE A CHEATING & VERIFIABILITY(Australian study with N=716 graduate applicants) Supervised Re-test CUT-OFF Online Test

18. Frequency 0 25 NMG4 at Assessment Centre (cut-off =25) Pre ONT Post ONT Equivalent to a saving of c.$2,700 per successful candidate Pass rate 30% to 50% (67%)

19. CONCLUSIONS • Main challenges for testing on the internet are: • Differentiating genuine tests from all the pseudo-tests • Technical, delivery issues • Security and authentication issues • Need to consider best practice in relation to different modes of administration • Open – low stakes and low security • Controlled – can be high stakes (if followed up), moderate security • Supervised or Managed– high stakes, high security • Need to consider how testing fits into the overall assessment process in order to manage content security, cheating and authentication. • Need international guidelines setting out best practice.

20. Thank you

22. Honesty policy and honesty management The management of test-taker honesty within a high stakes assessment process is not just a matter of supervision. It is also a matter of the design of the whole process and the extent to which cheating or dishonest behaviour is perceived as being likely to be detected. Assessment processes for selection can be backed by an explicit honesty policy to which candidates are asked to sign-up. This is supported by the process of re-assessing, under controlled supervised conditions in the final stages of the selection process, any key competencies which formed the basis for the sift. Such agreements are used to 'click-wrap' the application. In order to enter the application process, the applicant has formally to agree to a statement such as the following: “I understand that the results of this assessment may be used in the process of deciding whether my application will be progressed further. I understand that if it is progressed, I may be required to complete a similar assessment again under supervised conditions. I confirm that I will endeavour to complete the assessment in accordance with the instructions given. I will do so on my own, without seeking or accepting assistance or support from any other person or persons. I undertake neither to make copies of any of the questions that will appear on screen during this assessment nor to pass on to any other person any information about the content of the assessment.” At the end of the test they may be required to further acknowledge that: “I confirm that the responses given by me during the assessment were provided without the assistance of any other persons and without the use of any aids or devices other than those explicitly permitted by the instructions” While such 'contracts' are not legally binding or able to guarantee that the applicant has actually abided by them, they do help provide a clear set of expectations and explain that failure to abide by these conditions could have undesirable consequences.