250 likes | 356 Views
E-assessment: a risk-based approach to success. Dr Chris Ricketts, Sub-Dean (Teaching Enhancement), Faculty of Technology, University of Plymouth and Director of Assessment, Peninsula College of Medicine and Dentistry. Overview. history institutional strategy risk analysis
E N D
E-assessment: a risk-based approach to success Dr Chris Ricketts, Sub-Dean (Teaching Enhancement), Faculty of Technology, University of Plymouth and Director of Assessment, Peninsula College of Medicine and Dentistry
Overview • history • institutional strategy • risk analysis • examples of success • discussion
History (1) • over 20,000 students • 6 faculties • 6 sites over 150 miles • some use of CAA on local servers
History (2) • Needs analysis in April 1999 • 12 staff, 5 faculties, over 3,000 students • Strategic introduction of CAA(University-wide)
Strategy (1) Zakrzewski’s (1999) ‘Catherine-wheel’ model • Module > Department > Faculty > University We chose University-wide availability, but • controlled pilot use in modules • undertook risk analysis • roll-out to University
Strategy (2) What to pilot? • formative assessment • summative (in-course) assessment • end of year examinations • a variety of subject areas
Strategy (3) • Decision 1: A steering group which involved staff from all affected parts of the University • Decision 2: Academic Board support • Decision 3: Staff training and support • Decision 4: Easy introduction to students • Decision 5: Get student feedback - and act on it
Strategy (4) Decision 6: the big one! • Don’t treat computer aided assessments differently • management • quality processes • security
General experience • bed of roses?…not • we did prepare carefully and try to consider all risks
Risk analysis (Ricketts & Zakrzewski, AEHE 2005) Types of risk • Pedagogic (P) • Operational (O) • Technical (T) • Web-based (W) • Financial (F)
Risk analysis • Define the risks • Estimate likelihood • Estimate severity (who is affected?) • Concentrate on the severe problems • How to avoid • What if it happens?
Defining the risks • The literature can help, but… … these need to be YOUR risks, not someone else’s • Estimate likelihood • Estimate severity (who is affected?) • Concentrate on the real problems • How to avoid • What if it happens?
Risk example 1 • P1: Assessment method not integrated into the curriculum • Likelihood? M • Who affected? Students, Academic staff • How much? Module • When? Before assessment This was not a high severity risk for us.
Risk example 2 • O7: Module size too large for number of workstations available • Likelihood? H • Who affected? Students, Academic staff, Support staff • How much? University • When? Before assessment This was the most high severity risk for us.
Risk management example O17: Different invigilation requirements for CAA not recognized • Liaise with examinations office • Produce guidelines for invigilators • Ensure support team in exam includes technical support staff • Academic staff to be present at start of on-line examination
Issues • load testing the system for large-scalesummative assessments • link between student records and assessment databases • student computing mistakes • adequate computer facilities on all sites • wayward staff
Some findings (from Roy Lowry) • Introduction of formative tests each week after lecture • Students are motivated enough to use the system • Although requiring some effort to set up, the system can be re-used • CAA in a formative mode has a significant impact upon learning
Next step • Expanded to cover all material • Use of MC, MR, drop-down boxes and numeric questions • Used for the end-of-module test • 103 students in Babbage open access area • Students obtained their marks immediatelyafter they finished the test
Student use • 30% of students attempted all of the tests(average mark: 72%) • 65% attempted some of the tests(average mark: 53%) • 5% did not use the system(average mark: 45%)
Benefits Benefits for staff... • easier to give frequent feedback • no marking! Benefits for students… • more frequent feedback • instant marking • more self-assessment
“Cost effectiveness” • Thanks to James Wisdom (ELEN Conference 2000) for the following
Cost benefit analysis • Useful when benefits can be expressed in monetary termseg.Saving in staff time (hence money)etc. • Is this why we use CAA?
Cost effectiveness • Useful when outcomes cannot be expressed in monetary terms • looks at outcomes in relation to goal “on time, on budget, to quality” • Is this why we use CAA?
Pedagogic effectiveness • Learners learning,and learning better • Must be part of cost effectiveness • Is this why we use CAA??
Questions • Over to you