The development of international guidelines on computer-based testing and the internet

1. The development of international guidelines on computer-based testing and the internet Iain Coyne Department of Psychology I.J.Coyne@hull.ac.uk

2. Background • ITC aims to promote good practice in testing • ITC has previously produced testing guidelines • ITC identified a need to look at the issues surrounding CBT/Internet testing • Market and use is increasing • Technological sophistication is improving • What good practice issues should be considered?

3. Aims • To produce a set of internationally developed and recognised guidelines that highlight good practice issues in computer-based (CBT) and Internet-delivered testing. • To raise awareness among all stakeholders in the testing process of what constitutes good practice.

4. Aims • Scope: • Broad sense of tests and testing • Online & on-screen (part or whole testing) • Low and/or high stakes • Proctored and un-proctored testing • Used in different testing scenarios • Who for? • Test developers, publishers, users • Relevant to others (consultants, test-takers, national bodies, trainers etc.)

5. Process • Literature search • UK-based small survey • ITC Conference on CBT and the Internet • Consultation phase 1 and revisions • Consultation phase 2 and revisions • Current presentation • Conference workshop • Approval and dissemination

6. First phase • Literature search: • Review of existing references, policies and guidelines • BPS, EFPA, BSI, & ATP guidelines were highly relevant • Small survey: • Focused on internet delivered personality testing • 12 test publishers • Questionnaires and follow-up interviews • Identified good practice in testing

7. First phase • ITC Conference (2002): • Workshops, papers, posters and keynotes • 254 delegates from 21 countries • First draft (V.03) – March 2003 • Technology – aspects of CBT/Internet testing • Quality – of testing and testing materials • Control – of test delivery, authentication, prior practice • Security – of testing materials and test-taker data

8. Second phase • Consultation 1 – version 0.4 (February 2004) • Sent to ITC mailing lists & published on web site • 15 responses from 8 different countries • Included APA Internet Task Force report (Naglieri et al., 2004) – version 0.5 • Consultation 2 – version 0.6 (July 2004) • Same sample as before • 7 responses mainly from UK and USA

9. The design • High level issues (4 general issues) • second level specific guidelines • accompanying examples to relevant stakeholder • sometimes further elaboration • Relevant stakeholders • Test developers • Test publishers • Test users

10. Example • Provide appropriate levels of control over CBT and Internet testing • Detail the level of control over the test conditions • Test Developers • Document the hardware, software, and procedural requirements for administration of a CBT/Internet test. • Provide a description of the test-taking conditions required for appropriate CBT/Internet test administration. • Design the CBT/Internet test to be compatible with country-specific health and safety, legal, and union regulations and rules (e.g., time on task).

11. The future • Current presentation (August, 2004) • Final draft version (October, 2004) • Workshop at BPS Occupational Psychology conference (January, 2005) • Formal approval and launch (July, 2005). Web site - http://www.intestcom.org/