Does interoperability actually work? A case study carried out with Boon Low and Niall Barr

1. Does interoperability actually work? A case study carried out with Boon Low and Niall Barr Niall Sclater Manager, Educational Systems University of Strathclyde

2. The IMS Question and Test spec • For the exchange of questions and tests • Processes include: • Authoring • Storage • Administration • Rendering

3. The packages tested • A leading commercial CAA package (QuestionmarkPerception) • The two leading virtual learning environments (WebCT and Blackboard) • A lesser known but highly innovative rendering tool (Canvas Arena) • A university-built tool developed for testing IMS QTI (CETIS Rendering Tool)

4. Scope of the evaluation • Concentrates on the exchange of questions between systems • Question creation • Exporting • Importing

5. Question 1 Question 1: Which electronic component allows alternating current (AC) to flow but prevents direct current (DC) flowing? ·A transistor ·A resistor ·A capacitor [correct] ·An inductor

6. Feedback for question 1 • Transistors are used to amplify or switch both AC and DC currents, the correct answer is capacitor. • Resistors impede the flow of both AC and DC currents, the correct answer is capacitor. • Yes, capacitors prevent DC current flowing. • Inductors have high impedance for AC currents and low impedance for DC currents, the correct answer is capacitor.

7. Question 2 Which of these symbols represents a Field Effect Transistor (FET)? • 1 • 2 • 3 • 4

8. Criteria • Question creation – the ease with which the two model questions can be created and the degree of accuracy of question rendering • Exporting – the ease and accuracy with which the following applications export questions into the IMS QTI specification • Importing –the degree with which the suite of test questions can be imported and how accurately the imported questions are rendered.

9. Results of the evaluation Question authoring • Perception • q1 entered without difficulty • q2 not so easy but workarounds found • WebCT • q1 fine • q2 – feedback reduced to a single section for each answer • Blackboard • Feedback restricted to one section for the correct answer and one for the incorrect ones

10. Results of the evaluation (2) Exporting • Perception • Worked fine • WebCT • Requires use of a command line tool • Some minor errors in the exported QTI code • Blackboard • Questions exported within an IMS content manifest but in a proprietary format

11. Results of the evaluation (3) Importing • Perception • No problems • Blackboard • Didn’t work • WebCT • Content migration tool only for systems admins • Extremely difficult to use even for them • Rendering

12. Rendering: WebCT - Perception

13. Conclusions • VLEs very disappointing • None of the other systems had problems with q1 • Questionmark has been more rigorous than the VLE vendors in implementing QTI • But even it had some problems • Does this bode well for interoperability?

14. Conclusions (2) • Still in early days • Most problems easy to correct • Need cross-vendor collaboration • Stress-testing programmes • VLE vendors may leave QTI compliance to QuestionMark

15. Further info www.cetis.ac.uk www.sclater.com