1 / 43

Secure, reliable and effective institution-wide e-assessment

Secure, reliable and effective institution-wide e-assessment. Paving the way for new technologies. John Dermo, e-Assessment Adviser, Teaching Quality Enhancement Group, Sara Eyre, Head of IT Customer Services, Learner Support Services.

rusk
Download Presentation

Secure, reliable and effective institution-wide e-assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Secure, reliable and effective institution-wide e-assessment Paving the way for new technologies John Dermo, e-Assessment Adviser, Teaching Quality Enhancement Group, Sara Eyre, Head of IT Customer Services, Learner Support Services. 12th International CAA Conference: Research into e-Assessment, Loughborough University, 8th and 9th July 2008.

  2. University of Bradford • founded 1966 • around 10,000 students • multi-cultural learning environment, students from over 110 countries

  3. ranked highly for Graduate Employment; focus on the ethos of 'Making Knowledge Work'. • constantly investing in campus and facilities

  4. Has an Ecoversity vision, working towards: • sustainable education • a healthy environment • social well-being • a thriving economy

  5. 7 Academic Schools • Engineering, Design and Technology • Health Studies • Informatics • Life Sciences • Lifelong Education and Development • Management • Social & International Studies

  6. Pathfinder Project 2007-8 • HEA/JISC e-Learning Pathfinder Programme • “Embedding Support Processes for e-Assessment to ensure the reliable and secure large-scale implementation of CAA.”

  7. JISC Project 2008-9 • JISC Institutional Exemplar Project • “Integrating thin client systems and smart card technology to provide integrated, flexible, accessible and secure E-assessment.”

  8. Drivers: reasons for moving to online assessment • National Student Survey: significant levels of student dissatisfaction with traditional paper-based assessment methods and feedback mechanisms exist across the HE sector • HEFCE Strategy for e-Learning has as objectives for pedagogy, curriculum design and development: “Encourage the use of technology to enable electronic assessment [and] produce and disseminate models of good e-learning practice including assessment.” (HEFCE, 2005)  • UoB E-strategy (2004-9) • LTA strategy • Increased use of e-learning at UoB • Student expectations

  9. Historical background: CAA @ UoB • Carried out for several years • At first, by individual lecturers using a range of software eg Test Pilot, BB Assessment Tool, QMP version 3 • Dealt with on an ad hoc basis • With time, growing numbers • Need for a scalable, streamlined supported service and procedures

  10. Aims of the Pathfinder project • to enhance student learning through authentic and innovative approaches to assessment that are accessible, flexible and available on-demand; • extended range of applications of computer-assisted assessment; • expanded use of both formative and summative computer-assisted assessment; • system to support the effective large-scale implementation of computer-assisted assessment across the University. For more information about this project see: http://www.brad.ac.uk/lss/tqeg/projects/caa.php

  11. Pathfinder project: outcomes • Developing systems for CAA • Rules and regulations for CAA • Training materials and sessions, briefings • Promoting online assessment • Identify optimum use of learning spaces for CAA • Developing contacts within Schools • Networking within CAA community • Subject Specific examples • Item banking

  12. 0 TheAssessment Process

  13. 0 Authoring Stage Create questions and group into quizzes, tests, exams and surveys using a Windows PC (or occasionally a web browser)

  14. 0 Scheduling Stage Formative: instructors schedule through Blackboard Connector Summative: admin manage participants, groups and schedules through Enterprise manager

  15. 0 Delivery Stage Deliver assessments using a standard (or secure) browser, link from a secure webpage or Blackboard module.

  16. 0 Reporting Stage Instructors report on the results through reports in Enterprise manager or through BB gradebook.

  17. The server set-up FORMATIVE / LOW-STAKES students Low Stakes server (v. 4.3) shared repository staff staff Gatekeeper students QMLDAP Secure High stakes Server (v. 4.4) shared repository SITS ? SUMMATIVE / HIGH-STAKES Gatekeeper

  18. Use of QMP at UoB

  19. Training materials

  20. Training sessions

  21. Web-based support materials

  22. BB support organisation

  23. CAA Policy

  24. Standard written instructions

  25. Subject–specific examples

  26. The full summative CAA process

  27. SURVEY DATA 1: Affective Factors 2: Validity 3: Practicality 4: Reliability 5: Security 6: Pedagogy 130 undergraduates 6 different academic schools. Age 18 to 43 (mean = 21.26) 68% of the sample were male, 32% female 92% formative e-assessment, 41% online summative assessment (ie 33% both)

  28. SPEAQ Findings Negatives Randomised items from question banks are unfair (2.66) Technical problems make online exams impractical (2.77) Online exams test IT skills as well as subject knowledge (2.85) I’d rather do exams online because I am used to working online (2.85)

  29. SPEAQ Findings Positives No serious concerns about health and safety issues (3.82); Using a computer does not add to the stress of exams (3.54); CAA is not just a gimmick – it does add to learning (3.45) Goes hand in hand with e-learning (3.44) Grades are secure (3.43) Feedback from CAA helps students learn (3.41). Login security is OK (3.40), CAA is suitable for the student’s subject (3.39) Students expect online assessment at university (3.36). CAA allows more accurate marking (3.32)

  30. Feedback from SPEAQ

  31. Feedback from SPEAQ

  32. Feedback from SPEAQ

  33. Feedback from SPEAQ

  34. Feedback from SUTTL SUTTL Staff Use of Technology in Learning and Teaching Survey April-May 2008 – 143 staff members

  35. Feedback from SUTTL

  36. Feedback from SUTTL

  37. Feedback from SUTTL

  38. Feedback from SUTTL

  39. From staff Some comments From students

  40. Aims of the JISC project • A new 100+ e-assessment cluster being constructed using Sunray thin client technology instead of conventional PC-based assessment systems. • Linked with smart card identification to deliver innovative system for summative e-assessment that is scalable and economical. • suitable for large classes but yet easily manageable; responsive to academic staff needs; flexible enough to deal with a wide range of e-assessment types that can be delivered without needing elaborate installation procedures or additional software to control for security and access. • Initiatives across University to move from paper-based MCQs to online delivery  Anticipated significant increase in use

  41. Aims of the JISC project • Inclusive, personalisable for widening participation, accessible for our disabled students and also sustainable and environmentally friendly. • secure integration with other key University systems (e.g. student records, module registration, room bookings and examination scheduling), • When not in use for summative purposes will be able to deliver aspects of formative assessment as well as standard network services. For more information about this project see: http://www.brad.ac.uk/lss/jisc/

  42. Challenges in the JISC project Some major issues that have been raised include: • Ensuring accessible delivery for disabled students • Managing students who require extra time • Avoiding double booking rooms • Dealing with students who forget smart card or login password • Students whose account is banned • Students on franchised courses • Correct student module enrolment in student records • Need to hold initial grades until exam board meet before transfer into student records • Anonymous marking of word processed scripts • Keeping online exam scripts secure • Controlling the length of online exams • Archiving online exams • Equipment failure part way through an exam • Comfort and stress affecting student exam performance online compared to paper-based

  43. Thank Youj.dermo@bradford.ac.uk

More Related