ECDL Test Development and Validation Process

1. ECDL Test Development and Validation Process Paul Davis, Oxford University Computing Services Garry Cleere The European Computer Driving Licence Foundation Ltd.

2. Test development process CAA Conference

3. Syllabus Definition (Expert Opinion) CAA Conference

4. Soft Test Characterisation • Non-threatening test experience • Predominantly performance based • Straight forward language • No trick questions • Coherent outputs • Sustains the metaphor of the driving licence • Ample time CAA Conference

5. Forum score CAA Conference

6. Test characteristics – reallife CAA Conference

7. Forumduration of tests CAA Conference

8. Mean duration CAA Conference

9. Criticality (Core Item Identification) Exercises CAA Conference

10. ModelTestsExercise • SMEs create 4 streams of tests • Spread core items across streams • Provide a reasonable coverage in each test • Provide syllabus coverage across streams • Average number of items CAA Conference

11. REF. TASK ITEM MT 1 MT 2 MT 3 MT 4 3.1.1.1 Open (and close) a word processing application.  NE  NE NE  NE 3.1.1.2 Open one, several documents. 1 3.1.1.3 Create a new document (based on default, other available template). 1 1 1 1 ModelTests Grids Scheme CAA Conference

12. Item Count 36 Items • Item Count dictated by coverage requirement • Dictated by congruency and criticality in tests • 100% Syllabus coverage throughout ECDL tests • Balanced, discriminating test profiles for Syllabus V.4 • Tests Syllabus Version 4.0 rigorously • High reliability in tests andbetween tests • Item Count standardisation in all Modules CAA Conference

13. Result against completion time CAA Conference

14. Cut Score /PassMark • Systematic approach and Industry based cut score technique (modified Angoff Method) • Underpinned by expert opinion • Cut Score exercises carried out for each module • Borderline Candidate, or Minimally Competent Candidate (MCC) • Meaningful Cut Scores based on profile of ECDL tests derived from ECDL V4 Syllabus • Item Count standardisation in all Modules • An auditable, standard cut score process • Common cut score across all ECDL core level modules, (M1 - 7) CAA Conference

15. CutScoreExercises CAA Conference

16. ModelTestsCollationandBalancing CAA Conference

17. Characterisation Test Template Version 4.0 CAA Conference

18. Item / Test Development Dependant upon: • Syllabus • Soft Test Characterisation • CTT • ECDL-F Evaluations • Outcome Probability Grids (expected pass rate from question) CAA Conference

19. Item / Test Development CAA Conference

20. Expert Review of Items / Tests • Syllabus Congruence • Syllabus Coverage • Compliance with CTT • Appropriateness of Item format • Technical Accuracy • Language & Clarity • Text Content CAA Conference

21. Expert Review : ATES Systems • Test Presentation / Interface Features • Timing & Information • Appropriateness of ATES Test Content • Test Content • Marking Algorithm • Candidate Information and Guidance • Practice Tests • Documentation and Training Materials • ATES Technical Support • Reporting • Test Generation • Security Features CAA Conference

22. Perceptions of tests CAA Conference

23. Simulation In-Application Rich Simulation Web-based MCQ/HS MQTB First 12 12 1 3 3 Second 8 7 5 5 5 Third 5 4 11 6 3 Fourth 3 5 12 6 3 Fifth 1 1 0 9 15 Choice of test engine (V3) CAA Conference

24. Expert Review : Test Design / ATES Systems • ATES System Overview • ATES System Hardware Requirements • ATES Operating System Requirements • Other Software Requirements • Minimum Screen Resolution • Assistive Technologies • Screen Design and Visual Impairment CAA Conference

25. Assembly and Administration of PilotTests • Tests are made available for Pilot Test activity • Qualitative analysis • Test taker and administrator observation sheets / questionnaires • Test data requirements Remember: • Across 4 streams for 7 tests across 100 countries and many languages. CAA Conference

26. Pilot Tests Review Pilot Test results are reviewed by the ATES Evaluations Group. Any changes requested in the tests based on Pilot testing are then undertaken before the live certification tests are administered. To ensure statistical validity some pre-testing on the fly is used until results don’t change. Maintenance then takes over. CAA Conference

27. EmpiricalEvidenceEvaluation ECDL Tests are evaluated and monitored on an ongoing basis by the ATES Evaluations Group using recognized psychometric measurements. CTT Version 4.0 - Test behaviour statement CAA Conference

28. Ongoing Maintenance of Tests ECDL tests are monitored on an ongoing basis by the ECDL Foundation ATES Evaluations Group. Tests are updated in line with Syllabus revisions and also where test results or empirical evidence indicates that items or tests are not functioning in line with test behaviour parameters. CAA Conference

29. ProcessStepsandtheValidityArgument • Documented process - expert opinion • Detailed test data reporting requirements • Data fed back to the ECDL Foundation • Industry Standard techniques • Process Loop • Valid ECDL Tests CAA Conference

30. Contacts - Garry Cleere Programmes Manager ECDL Foundation 107 The Windmill Sir John Rogersons’ Quay Dublin 2 Ireland Tel. 00 353 1 679 2847 Tel. 00 353 1 679 2921 garry.cleere@ecdl.com • Dr Paul Davis • Deputy Head, LTG, • OUCS • 13 Banbury Road • Oxford • OX2 6NN • UK • Tel: 01865 283414 • paul.davis@oucs.ox.ac.uk CAA Conference