Summative Assessments Online: High Stakes – Low Risks

1. a Summative Assessments Online: High Stakes – Low Risks Dr Richard Parsons Centre for Learning and Teaching University of Dundee r.parsons@dundee.ac.uk

2. University of Dundee • Considerable experience of online assessments • Blackboard VLE users • Delivering up to 2000 formative assessment sessions a day with QM Perception • Delivering 50+ summative exams a year with QM Perception • Further expansion expected

3. Online Assessment System Assessment hardware design & deployment Assessment software design & deployment Clear policy and procedures Central assessment support Learning systems integration Staff and student education

4. The Hardest Task Remains writing quality questions • Staff time is precious • Students are more critical than ever before • Assessments to be aligned • Questions need to be prepared ahead of time • Quality runs to feedback, distracters, variety, depth, currency • Authoring Q within the software is secondary

5. Staff and Student Education • At Dundee we have an Online Assessment course • Covers technical and pedagogical aspects • 5 weeks long, 3 to 6 hours a week. • Formally accredited by our Education Faculty (10 SC) • Building a bank of staff skills • Most students need little guidance

6. Examples of Assessment Strategies • 15 Q from pool of 50, max mark taken (Languages) • Take assessment first, then repeat in an open fashion. Mark taken from first run. (Medicine) • Revision questions with feedback relating to lectures (Art History) • 10 numerical questions, each from a pool of 50 randomly generated Q (Life Sciences) • Student revision – students invited to compose 2 Q each with FB, all compiled automatically, checked & made available for open assessment (Env Sci) • Open assessment, one submission only (Law) • Summative (range of departments)

7. Quality Hardware and Systems • Central servers built & maintained by ITS • LEU involved in system design • Dual servers, one formative, one summative • Compaq ML370 Dual Pentium 1400 MHz 2GB RAM, single NICs • Databases • Formative on Oracle Unix server • Summative on local Access

8. LSExam2 Backup route only CAA-02 xx_username Matric no. Existing & operational Operational Sept 2003 QMP formative CAA server CAA-02 Blackboard VLE – ELS 6.0 14000 student users, 1000 staff users QuestionMark Perception application and database server Windows 2000 Server, IIS, Oracle 8i database Compaq ML370 Dual PIII 1400MHz 2 GB RAM 200 GB RAID5 array Blackboard VLE web application server – Dumpty Sun V480 application server 4 pro, 16GB RAM 800GB + RAID5 array Database server Oracle 8i. Humpty Sun V480 database server 4 pro, 16GB RAM 800GB + RAID5 array Assessment publication CAA-01 Database server Oracle 8i. LEU (DJW & RP PCs) linked by Netdrive and Oracle 8i client Quality Hardware Development CAA system Development VLE environment QMP Test CAA server CAA-03 Blackboard VLE – ELS 6.0 Test Blackboard VLE server replica of production system Sun Blade 1000, 2 750MHz processors, 2 GB RAM A1000 300GB RAID5 array Existing & operational QMP examination CAA server CAA-01 CAA-01 Username Matric no. QuestionMark Perception application and database server Windows 2000 Server, IIS, Access database Compaq ML370 Dual PIII 1400MHz 2 GB RAM 200 GB RAID5 array LSExam1 Assessment publication LEU (DJW & RP PCs) linked by Netdrive and Access publish tool

9. Clear Policy and Procedures • Central online assessment system required clear guidance • Prepared draft Policy and Procedures • Generally follows BS7988:2002 • Developed for about a year, inc. consultation • Now passed at University Senate • Excellent for ensuring quality • Runs to 16 pages • Additional documentation for Online Assessment system administrators

10. Summative Procedures • Assessment must be tested • Assessment loaded to both servers, scheduled on one • Central staff attend examinations • Use special ”exam” accounts • Use Perception Secure Browser • Candidates complete Examination Attendance form • Immediately after the exam a backup of results is taken • Use two way radios or mobiles • No system is fullproof, 99% success target ?

17. Final Points of Interest • Once had a problem of results displayed immediately to students finishing a summative • Industrial action on assessments left the machines running unsupported • The best people to assess a subject are experts in the subject, not experts in online assessment • OA is appropriate for usage outside of the sciences • Staff are becoming “managers of teaching” • Online assessment is important, but not life threatening

18. The Future for Online Assessment • Secure, stable, performing system • Increased range and depth of questions • Innovation within and beyond QM • Work with the staff on questions • Increased sharing/sales of question banks • Foresee increased integration with online learning systems

19. Summary • Dundee has an institutional online assessment system • Software, hardware, P&P, staffing, integration and education • Secure and reliable, but straightforward to use for students and staff • Let the academics create the Qs to stimulate learning • Costs are neither insignificant or astronomical • Expect the internet developments to continue

20. University of Dundee Managed Learning Environment January 2004 GroupWise Web Email Student Management System P3 QM Formative server CAA-02 User exists 3 weeks before appointment Completed daily Module enrolment Bb client Mark return to Bb Student exists Staff details also stored in SMS Module enrolments of students Module assessment mark storage LDAP client? Blackboard VLE – ELS 6.0 QM Exam server CAA-01 Hardware 2 * 480s Version 6.0.10 Users created from SITS SMS system Module enrolment Staff assignments from web system LDAP client Module assessment mark storage Training teaching staff Special exam enrolment Mark return Library Catalogue access Resource management system LDAP client? e-Directory LDAP Dual Novell LDAP servers Create user name and email process LDAP authentication Remove user process Staff Assignment Record File store Novell Fileservers Student Desktop Secure web access Richard Parsons, CLT LEU