Secure, reliable and effective institution-wide e-assessment

1. Secure, reliable and effective institution-wide e-assessment Paving the way for new technologies John Dermo, e-Assessment Adviser, Teaching Quality Enhancement Group, Sara Eyre, Head of IT Customer Services, Learner Support Services. 12th International CAA Conference: Research into e-Assessment, Loughborough University, 8th and 9th July 2008.

2. University of Bradford • founded 1966 • around 10,000 students • multi-cultural learning environment, students from over 110 countries

3. ranked highly for Graduate Employment; focus on the ethos of 'Making Knowledge Work'. • constantly investing in campus and facilities

4. Has an Ecoversity vision, working towards: • sustainable education • a healthy environment • social well-being • a thriving economy

5. 7 Academic Schools • Engineering, Design and Technology • Health Studies • Informatics • Life Sciences • Lifelong Education and Development • Management • Social & International Studies

6. Pathfinder Project 2007-8 • HEA/JISC e-Learning Pathfinder Programme • “Embedding Support Processes for e-Assessment to ensure the reliable and secure large-scale implementation of CAA.”

7. JISC Project 2008-9 • JISC Institutional Exemplar Project • “Integrating thin client systems and smart card technology to provide integrated, flexible, accessible and secure E-assessment.”

8. Drivers: reasons for moving to online assessment • National Student Survey: significant levels of student dissatisfaction with traditional paper-based assessment methods and feedback mechanisms exist across the HE sector • HEFCE Strategy for e-Learning has as objectives for pedagogy, curriculum design and development: “Encourage the use of technology to enable electronic assessment [and] produce and disseminate models of good e-learning practice including assessment.” (HEFCE, 2005)  • UoB E-strategy (2004-9) • LTA strategy • Increased use of e-learning at UoB • Student expectations

9. Historical background: CAA @ UoB • Carried out for several years • At first, by individual lecturers using a range of software eg Test Pilot, BB Assessment Tool, QMP version 3 • Dealt with on an ad hoc basis • With time, growing numbers • Need for a scalable, streamlined supported service and procedures

10. Aims of the Pathfinder project • to enhance student learning through authentic and innovative approaches to assessment that are accessible, flexible and available on-demand; • extended range of applications of computer-assisted assessment; • expanded use of both formative and summative computer-assisted assessment; • system to support the effective large-scale implementation of computer-assisted assessment across the University. For more information about this project see: http://www.brad.ac.uk/lss/tqeg/projects/caa.php

11. Pathfinder project: outcomes • Developing systems for CAA • Rules and regulations for CAA • Training materials and sessions, briefings • Promoting online assessment • Identify optimum use of learning spaces for CAA • Developing contacts within Schools • Networking within CAA community • Subject Specific examples • Item banking

12. 0 TheAssessment Process

13. 0 Authoring Stage Create questions and group into quizzes, tests, exams and surveys using a Windows PC (or occasionally a web browser)

14. 0 Scheduling Stage Formative: instructors schedule through Blackboard Connector Summative: admin manage participants, groups and schedules through Enterprise manager

15. 0 Delivery Stage Deliver assessments using a standard (or secure) browser, link from a secure webpage or Blackboard module.

16. 0 Reporting Stage Instructors report on the results through reports in Enterprise manager or through BB gradebook.

17. The server set-up FORMATIVE / LOW-STAKES students Low Stakes server (v. 4.3) shared repository staff staff Gatekeeper students QMLDAP Secure High stakes Server (v. 4.4) shared repository SITS ? SUMMATIVE / HIGH-STAKES Gatekeeper

18. Use of QMP at UoB

19. Training materials

20. Training sessions

21. Web-based support materials

22. BB support organisation

23. CAA Policy

24. Standard written instructions

25. Subject–specific examples

26. The full summative CAA process

27. SURVEY DATA 1: Affective Factors 2: Validity 3: Practicality 4: Reliability 5: Security 6: Pedagogy 130 undergraduates 6 different academic schools. Age 18 to 43 (mean = 21.26) 68% of the sample were male, 32% female 92% formative e-assessment, 41% online summative assessment (ie 33% both)

28. SPEAQ Findings Negatives Randomised items from question banks are unfair (2.66) Technical problems make online exams impractical (2.77) Online exams test IT skills as well as subject knowledge (2.85) I’d rather do exams online because I am used to working online (2.85)

29. SPEAQ Findings Positives No serious concerns about health and safety issues (3.82); Using a computer does not add to the stress of exams (3.54); CAA is not just a gimmick – it does add to learning (3.45) Goes hand in hand with e-learning (3.44) Grades are secure (3.43) Feedback from CAA helps students learn (3.41). Login security is OK (3.40), CAA is suitable for the student’s subject (3.39) Students expect online assessment at university (3.36). CAA allows more accurate marking (3.32)

30. Feedback from SPEAQ

31. Feedback from SPEAQ

32. Feedback from SPEAQ

33. Feedback from SPEAQ

34. Feedback from SUTTL SUTTL Staff Use of Technology in Learning and Teaching Survey April-May 2008 – 143 staff members

35. Feedback from SUTTL

36. Feedback from SUTTL

37. Feedback from SUTTL

38. Feedback from SUTTL

39. From staff Some comments From students

40. Aims of the JISC project • A new 100+ e-assessment cluster being constructed using Sunray thin client technology instead of conventional PC-based assessment systems. • Linked with smart card identification to deliver innovative system for summative e-assessment that is scalable and economical. • suitable for large classes but yet easily manageable; responsive to academic staff needs; flexible enough to deal with a wide range of e-assessment types that can be delivered without needing elaborate installation procedures or additional software to control for security and access. • Initiatives across University to move from paper-based MCQs to online delivery  Anticipated significant increase in use

41. Aims of the JISC project • Inclusive, personalisable for widening participation, accessible for our disabled students and also sustainable and environmentally friendly. • secure integration with other key University systems (e.g. student records, module registration, room bookings and examination scheduling), • When not in use for summative purposes will be able to deliver aspects of formative assessment as well as standard network services. For more information about this project see: http://www.brad.ac.uk/lss/jisc/

42. Challenges in the JISC project Some major issues that have been raised include: • Ensuring accessible delivery for disabled students • Managing students who require extra time • Avoiding double booking rooms • Dealing with students who forget smart card or login password • Students whose account is banned • Students on franchised courses • Correct student module enrolment in student records • Need to hold initial grades until exam board meet before transfer into student records • Anonymous marking of word processed scripts • Keeping online exam scripts secure • Controlling the length of online exams • Archiving online exams • Equipment failure part way through an exam • Comfort and stress affecting student exam performance online compared to paper-based

43. Thank Youj.dermo@bradford.ac.uk