Reflections on innovations in e-assessment

1. Reflections on innovations in e-assessment John Winkley May 2014

2. About AlphaPlus • Specialist education and assessment consultancy • Particular strength in qualifications, assessment and examinations • Extensive work in technology-supported learning and assessment • Based in the UK, with an international portfolio of work, covering • Primary, secondary and high school examinations for progression to college university • Vocational and technical qualifications • Professional certifications

3. About me! • Started out as an engineer/technologist – two organisations providing e-assessment and e-learning. • www.btl.com (millions of online exams each year) • www.virtual-college.co.uk (recently enrolled its 1.3 millionth learner) • Then moved in to e-assessment, and then assessment more generally. • It’s not really about the technology any more… • I work with educators, test writers, certification agencies and governments to help make qualifications and assessment work better

4. Potential of e-assessmentfor large-scale testing Bennett (2001) In the same way that it is already helping to revolutionize commerce, education, and even social interaction, the Internet will help revolutionize the business and substance of large-scale assessment.

5. But has potential gone unrealised? Isabel Nisbet, outgoing chief executive of Ofqual February 2011

6. Isabel in text ‘The way that students are tested should be the same as the way that they learn, otherwise it’s not a very good test… ‘…the way that they use technology to learn that that’s just native to them… …sitting at desks writing down, that may not be testing what they’ve learnt. ‘…let’s get this off the too difficult pile and just think a little bit about the future, how are we going to make sure that tomorrow’s kids have the right kind of tests?’

7. Ofqual in 2013 Jeremy Benson, Director of Policy, Ofqual at FAB national conference 2013 The starting point for my speech today is standards in vocational qualifications. There has [not] been … nearly enough [talk] about standards. But in my view, standards are as important in vocational qualifications as in general qualifications. Standards covers not only the skills that have to be demonstrated to pass the qualification, but also the level of demand from the design and structure of assessments, and how well candidates have to perform. How can we, as regulator, be confident that your assessment developers and your assessors have the right skills, so you can be confident in the standards and validity of your awards? Where will the assessors of the future come from? – particularly in an increasingly e-enabled world?

8. Focus of this talk • Maybe high-stakes testing in schools isn’t where e-assessment can have the most (immediate) benefit. • Maybe we can get more ‘wins’ from non-testing, less ‘schools’ technologies and services. • Some “case studies”: • Changes in “traditional e-assessment” • On-screen essays • Elephants don’t forget! • Secure exam delivery by pdf • Low volume marking • Statistical services for AOs • I will illustrate the case studies and explain why they are successful (or at least interesting) e-assessment innovations.

9. The case studies

10. CASE STUDY 1: on screen tests in 2014 Actually, a few disparate experiences from various projects‘it’s really not about the technology any more’ Our own experience in deploying e-assessment for a major project for BIS Our experience in running a training course recently in Saudi Arabia Falling price of e-assessment “per-test”. Falling minimum volumes too. Cloud-based delivery appears to have solved a lot of problems for centres

11. CASE STUDY 2: on screen essays The brief: Specialist professional body offering international examinations Small volumes, limited internal resources – previous investigations of “traditional” e-assessment not fruitful But board level drive to undertake service modernisation to support growth Examinations take the form of lengthy essay-based papers. Changing the nature of the examination was outside the scope of the project. The administration involves moving a lot of paper around the world. Candidates are typically mid 30s busy professionals working in large companies

12. CASE STUDY 2: on screen essays What we helped the professional body to do: Review the options in the market including some of the “less trodden paths” Deploy a “Bring Your Own Device” e-assessment solution using the Exam4 “armored word processor” Support centres and learners participating in the live pilot Obtain detailed feedback on how they found the assessment Deploy a secure exam paper delivery system Support centres using the technology

15. CASE STUDY 2: on screen essays What we learned: Students really valued being able to use their own machine Familiarity Able to type-and-edit faster than they could write Removed concerns about untidy and illegible scripts Many students were willing and able to work through the setup in advance Technology effective and inexpensive “Start to finish” in about 6 months

16. CASE STUDY 3: Elephants don’t forget Employer ownership pilots are showing interest in unfamiliar approaches to assessment and “awarding”, particularly in areas where employers have existing assessment (usually internal compliance or safety-related) which is similar to external assessment. • Seems likely that BIS will want Ofqual to consider “approval” of such innovative approaches. • Despite the jokey website, this approach could potentially deliver very interesting effects in terms of assessment predictive validity (we think).

17. CASE STUDY 4: Secure exam delivery Awarding organisations: • Upload Exam Packs to the system. An Exam Pack contains all the PDF files you want the centre to be able to download in advance of the examination. • Create Examination Windows. An Examination Window specifies when an Exam Pack becomes available for download, and when the download window closes. • Allocate the Examination Window to one or more Centres. • Mass email the Centres (via SecurePDF) to let them know about their upcoming Examination Windows.

18. Monitor the status of downloads

19. How does it work for centres?

20. CASE STUDY 4: Secure paper delivery What we learned: • Greatly increased security for delivery of examination documents (versus postal delivery or website/portal download) • Awarding organisation has complete control over when centres download examination documents • SecurePDF maintains a record of downloads, identifying users and centres that downloaded each Exam Pack • Centres were happy to use the system and valued the efforts made to improve security

21. CASE STUDY 5: Low volume marking There are four key problems when it comes to marking digital files: How do we view all the different file types without lots of software? Files can be very large – e.g. PowerPoint How do we actually mark/annotate the files without changing the work? How do we track different users’ annotations clearly?

22. Red pen – how it works The Benefits: Mark and annotate a variety of file types completely online No need for a multitude of software - just a Flash plug-in Does not alter the work Links to mark schemes Provision for student self, peer, teacher and verifier assessment.

23. Case Study – Red Pen Tool for Moodle Oldham Hulme Grammar School • Using Moodle to support work at KS3 & 4 • Found that there was still too much printing, especially for marking work • They started using Red Pen Tool in February 2011 • Primarily using the Red Pen Tool to support coursework for GCSE ICT (Module 2: controlled assessment - WJEC) • Proved to be a huge success and as a result they are planning to extend use this coming year.

24. CASE STUDY 4: Statistical analysis for AOs AOs who use on-screen tests will have large quantities of data. It is possible to use such data to do a wide range of analysis. Such analyses are very useful as a ‘reality check’. Helps you understand tests’ performance empirically. “Experts consider this item difficult, but when we presented it to 5,000 students, the facility value was 0.75, so it's pretty easy.” Statistical analysis does not replace expert judgement, it supplements it. It’s part of a general commitment to quality. Helps AOs to spot problems before anyone else does! There are ‘pushes’ as well as ‘pulls’ Ofqual increasingly looking into this area. Likely to ask ‘do you know how reliable your tests are?’ Parallel forms an area of particular weakness, we believe

25. Item difficulty vs ‘curriculum level’

26. Reflections on innovations in e-assessment Conclusions

27. Conclusions Lower cost, lower volume solutions ever more viable for organisations that would not consider themselves as experts in IT The view of on-screen assessment as a commodity is becoming more prevalent (and more true) but is not the whole story Diversity of options increasing too. Careful market searches identify tools that solve different problems to the mainstream products Technology offers AOs proven competitive advantage, at a time of extraordinary market turbulence (for the Ofqual regulated sector)

28. Reflections on innovations in e-assessmentThank you for listening.Questions? John Winkley May 2014