240 likes | 345 Views
a. A University Strategy for Delivering Online Assessment. Dr Richard Parsons Director of the Learning Centre University of Dundee r.parsons@dundee.ac.uk. University of Dundee. More than 10 years substantial experience of online assessments
E N D
a A University Strategy for Delivering Online Assessment Dr Richard Parsons Director of the Learning Centre University of Dundee r.parsons@dundee.ac.uk
University of Dundee • More than 10 years substantial experience of online assessments • Blackboard clients – Academic Suite (virtually 100% modules) • Delivering up to 3500 formative assessment sessions a day with QM Perception • Delivering 50+ summative exams a year with QM Perception • Range of online assessment activities recognised
Blackboard Server Configuration Summer 2006 – Bb upgrade 6.3 to 7.1 on Unix, QM application from 3.4 to 4.1, Bb - QM bridge software (twice), QM repositories for assessments, Blackboard building block deployments and upgrades, Solaris and Oracle database upgrades, Annual module rollovers, Enhanced PDP programmes, Medical School curriculum map integration, University restructuring configuration changes
Online Assessment System Assessment hardware design & deployment Assessment software design & deployment Clear policy and procedures Central assessment support Learning systems integration Staff and student education
The Hardest Task Remains writing quality questions • Staff time is precious • Students are more critical than ever before • Assessments to be aligned • Questions need to be prepared ahead of time • Quality runs to feedback, distracters, variety, depth, currency • Authoring Q within the software is secondary
Staff and Student Education • For 4 years at Dundee we have run an Online Assessment course, online • Covers technical and pedagogical aspects • 5 weeks long, 3 to 6 hours a week. • Formally accredited by our Education Faculty (10 SC) • Building a bank of staff skills • Most students need little guidance
Examples of Assessment Strategies • 15 Q from pool of 100, max mark taken (Genetics) • Take assessment first, then repeat in an open fashion. Mark taken from first run. (Medicine) • 10 numerical questions, each from a pool of 50 randomly generated Q (Life Sciences) • Student revision – students invited to compose 2 Q each with FB, all compiled automatically, checked & made available for open assessment (Env Sci) • Open assessment, one submission only (Law) • Summative (range of departments)
Quality Hardware and Systems • Central servers built & maintained by ITS • LC involved in system design • Three servers, one formative, one summative, one shared repository • HP BL20p Blades Dual Pentium 3400 MHz 6GB RAM, dual NIC backplane. • Databases • Formative on Oracle Unix server • Summative on Oracle Unix server • Shared on Oracle Unix server
University of Dundee Questionmark Perception V4 Systems Model Participant Participant Oracle 10G Oracle 10G QMP V4.2 CAA-02 Production Formative Repository Participant QMP V4.2 CAA-01 Production Exam Repository Participant LC Administrator QM / Bb integration QMP V4 Authoring Manager Oracle 10G Oracle 10G Participant QMP V4.2 CAA-03 DevelopmentRepository Blackboard 7.1 My Dundee Local Author (Staff) Participant QMP V4 Authoring Manager Participant Local Author (Staff) QMP V4 Authoring Manager Local Author (Staff) David Walker, University of Dundee
Clear Policy and Procedures • Central online assessment system required clear guidance • Prepared draft Policy and Procedures • Generally follows BS7988:2002 • Developed for about a year, inc. consultation • Passed at University Senate • Excellent for ensuring quality • Runs to 17 pages • Additional documentation for Online Assessment system administrators
Summative Procedures • Assessment must be tested • Assessment loaded to both servers, scheduled on one • Central staff attend examinations • Use special ”exam” accounts • Use Perception Secure Browser • Candidates complete Examination Attendance form • Immediately after the exam a backup of results is taken • Use two way radios or mobiles • No system is fullproof, 99% success target ?
Self & Peer Assessment Bb Building Block • Assessments are added and edited by the instructors in any course • One or more questions, each with one or more marking criteria • Marking criteria as discrete values or range • Feedback provided as score and text if required • Students automatically allocated to mark peers (1 to many) • Peer assessments are assigned automatically • Self assessment is an option • Marks and reports are visible to the Instructor • All marks can be downloaded for analysis in excel • Moderated results automatically passed to Bb Gradebook • Exercise text is available for download and online plagiarism checks • Questions and criteria are stored for reuse and exchange
The Dundee My PDP resources Online resource to support PDP; timetabled induction sessions; scope for faculty tailoring.Available for free under license. Each student can prepare an online portfolio based on templates, culminating in prioritised action points and an enhanced CV. Portfolios can be shared online.
Mobile Campus Map • Java game-based map • Rich links with key campus information • Single download • Popular with students www.dundee.ac.uk/mobilemap
Additional Online Assessment • Self & Peer • Framework for plagiarism advice and management • Procedures for online submission • Discussion board assessments • ePortfolio assessment rubrics and tools • Classroom/personal response systems • Business intelligence activities • Overall framework of OA support at Dundee
Individual concerns within an online assessment world Traditional Lecturer Who can electronically author my questions?Can we use textbook questions?Is this assessment driving learning? College Officer Can we book an IT suite for 240 students on Wednesday the 25th?Two students have additional requirements.Will the results transfer directly to the SRS? eLearning Manager Are the staffing levels correct?Is our software choice appropriate?Are the server specifications correct to cope with the expected load?Are staff education procedures in place? Lecturer Are my questions too simple?Is this technology reliable?Do the students find it tedious?Where can we obtain challenging questions?Are these assessments aligned with our learning outcomes? IT Desktop Administrator Does the desktop software operate correctly?Are the “exam accounts” secure and available?Are the PCs well maintained and reliable? Timetabling Officer Can the proposed exams be fitted into the best rooms?Are assessments receiving appropriate prioritisation? Student This is nice and privateI am getting real feedback hereDo I know the answers?Can I learn from this?I am at home in this online environment Distance Student Is my technology compatible?Is my connection reliable?Is this a replacement for F2F? IT Server Administrator Is the server fully patched, reliable and secure?Is the OA software functioning correctly and reliablyDo we have monitoring and is it working?Are the logging and backup procedures working correctly? Online Assessment Officer Are all assessments up-to-date and correctly available?Is the server-client functionality all compatible?Will the next examination run smoothly?Are the staff education programmes ready? Examinations Officer Are these results 100% reliable?secure?Do our procedures represent the best possible practice? Richard Parsons, University of Dundee
Elements Actors Procedural Each addressed by a tasked individual(s) within a supported team, and each element able to cope with growth and changes Revisit annually? The entire operation has limited inertia, limited history, sustainable staffing, space, budget. Represents a framework OA policy and procedures ZZ + AA Student and staff training XX + WW Examination practices Academic staff + XX + WW Room booking procedures College staff Student personal PCs Web guidance + VV University student desktop BB + CC Question creation Academic staff + XX + WW Question/assessment repositories XX + WW Network DD + EE OA application software & support XX + WW 13 specialist staff + academics & college staff Technical database considerations FF + GG Operating system & web server XX + HH Hardware & infrastructure HH + II Technical Richard Parsons, University of Dundee
The Annual Cycle of Online Assessment Sept. Testing & tuning Modules begin Online resit exams Annual frameworkmeeting Summer upgrade window Online exams June Dec. Mid semester window Online exams • Growth may occur each year • Changes are made to the systems • A framework allows the process to be sustained OA staff education module Richard Parsons, University of Dundee March
Summary • Dundee has an institutional online assessment & some elements of a strategic framework • Software, hardware, P&P, staffing, integration and education. 13 specialist staff. • Secure and reliable, but straightforward to use for students and staff • Let the academics create the Qs to stimulate learning • Costs are neither insignificant or astronomical