Developing a Roadmap for e-Assessment: Which way now?

1. Developing a Roadmap for e-Assessment: Which way now? Denise Whitelock and Andrew Brasher Institute of Educational Technology The Open University, e-mail: d.m.whitelock@open.ac.uk DMW 2006 The Open University's Institute for Educational Technology

2. Acknowledgements Sarah Knight, JISC Paul Bailey, JISC Andrew Brasher, The Open University Simon Rae, The Open University All the experts surveyed in the study DMW 2006 The Open University's Institute for Educational Technology

3. Background • e-Assessment is of strategic importance to the UK • Major global growth industry • Recognised by JISC’s e-Learning Programme • Funded by JISC to produce a Roadmap DMW 2006 The Open University's Institute for Educational Technology

4. What is a roadmap? • Obtain a consensus view or vision of the future landscape of e-assessment to decision makers • Kostoff & Schaller (2001) • roadmapping provides a way to identify and evaluate and select strategic alternatives for the desired objective DMW 2006 The Open University's Institute for Educational Technology

5. Road mapping • Technology push • Requirements pull • Layers • Know why – envisioning activities • Know what – policy and theoretical developments • Know how - technological • Time Dimension • Adapted from Phaal, Farrukh et al, 2004 DMW 2006 The Open University's Institute for Educational Technology

6. Adapting the Delphi Method • Delphi questionnaire often used http://www.is.njit.edu/pubs/delphibook/index.html#toc • Consensus • Reduced to lowest common denominator • No priorities emerge • This consultation will identify core and outlying issues – avoid polarities DMW 2006 The Open University's Institute for Educational Technology

7. Application of the Delphi Method DMW 2006 The Open University's Institute for Educational Technology

8. Building the e-Assessment Roadmap • Map current terrain • Review current policy and initiatives • Identify current practice • Recognise practitioners’ concerns • Check whether rhetoric matches reality • Gauge stakeholders’ visions • Recommendations for keeping on track DMW 2006 The Open University's Institute for Educational Technology

9. 3 stage build of the Roadmap DMW 2006 The Open University's Institute for Educational Technology

10. Roadmap framework DMW 2006 The Open University's Institute for Educational Technology

11. Roadmap framework linkages DMW 2006 The Open University's Institute for Educational Technology

12. Sourcing the information for Stage 1 • Superinstitutional • Policymakers : HEFCE, SQA etc. • Institutional • HE, FE etc. • Personal • Tutors, students, developers DMW 2006 The Open University's Institute for Educational Technology

13. Superinstitutional: Policies and Practices • e-Assessment will be rolled out in post-16 education by 2009 • e-Assessment will make a significant contribution to reducing the assessment burden and improving the quality of assessment • e-Assessment field trials should be taking place in at least two subjects per awarding body during 2005 • 75% of key and basic skills tests will be delivered on screen by 2005 • N.B. 2009 deadline made explicit by QCA but not SQA DMW 2006 The Open University's Institute for Educational Technology

14. Personal: Visions from Focus Groups • Complete shift from paper exams • Acceleration of evidence gathering tools for assessment • Video assessment of key skills for NVQs • Exam Boards move to paperless and machine marking exams • Real life simulations to test medics and lawyers at University • Professional Bodies will change to e-assessment DMW 2006 The Open University's Institute for Educational Technology

15. Personal: Barriers from Focus Groups • More training needed • Better plagiarism detection • Exam centres should be in place • Technical expertise not always available in schools • Computers could be used solely for examinations if just in time testing available. Therefore unavailable for lesson use in schools DMW 2006 The Open University's Institute for Educational Technology

16. Consulting Experts with a Survey : Stage 2 • Survey designed to probe the following: • Timings of policy implementation • How e-Assessment can cut the burden of assessment • Ways in which the quality of e-Assessment will be improved • Implications of the vision of the policy documents • Visions for the future • Barriers to the visions DMW 2006 The Open University's Institute for Educational Technology

17. Experts • Superinstitutional • Representatives from SQA, HEFCE, BECTA etc. • Institutional • Directors of e-Learning Units and policy makers in HE and FE • Personal • Champions of e-Assessment DMW 2006 The Open University's Institute for Educational Technology

18. Survey Findings (1) • Predicted timings: • Agree with 2010 deadline • ICT accepted in all aspects of student experience within 2/4 years • Tutors will have tools to assist with course design and providing electronic feedback to students within 2/4 years DMW 2006 The Open University's Institute for Educational Technology

19. Survey Findings (2) • Cutting the burden • Technological change means revamp of old practice • Questions need to be more interactive • Efficiencies is in results processing, transparency in grading, develop criterion referenced exams • Introduce formative assessment • Faster feedback • Better plagiarism detection • Production of essay marking tools • Accessibility issues require higher profile DMW 2006 The Open University's Institute for Educational Technology

20. Survey Findings (3) • Improving the quality of e-Assessment • Offer more realistic assessments • Cover wide range of curriculum • Forces assessors to produce clearly defined objectives that enhance student learning • Better regular feedback will assist students who usually under-perform • More improvement seen in vocational sphere DMW 2006 The Open University's Institute for Educational Technology

21. Survey Findings (4) • Implications for the vision set by the policy documents • e-Assessment sites will be open 24/7 • More students will study University courses while at school • Parents will over-pressurise students to take exams too early • Over-use of on-demand testing does not always increase grades and produces lack of confidence in standards • e-Assessment to play key role in personalised learning agenda DMW 2006 The Open University's Institute for Educational Technology

22. Policy implications • 50% agreed students studying at own pace will result in higher grades and a lack of confidence in exam style • Agreed resits should be regulated • Agreed more breadth and depth of learning will be achieved through e-assessment • 16 – 18 years will be studying some university courses while at school DMW 2006 The Open University's Institute for Educational Technology

23. Paperless examinations as recommended by CCEA and Edexcel • Authors need training • Authors do not need same technical skills as technology partner • Research into different question types sorely needed • Makes work at home on own screen • Good piloting essential DMW 2006 The Open University's Institute for Educational Technology

24. Further Expert Opinions • Adaptation of existing standards regarding security (BS7799), quality (ISO9000), test delivery (BS7988) and item production (QTI). Commercial developers will need to subscribe to these standards • Strong encouragement of open-source software licensing models • An agreed level of interoperability based on national standards. DMW 2006 The Open University's Institute for Educational Technology

25. Visions: Superorganisational • Large-scale testing sites • Test sites will deliver some paper and pencil components • Prevalent from primary through to university and vocational qualifications • High quality question banks • No longer referred to as e-assessment DMW 2006 The Open University's Institute for Educational Technology

26. Visions: Organisational • e-Portfolios increase • Formative, self assessment and e-portfolios main tools • Random internal testing • Confidence declared in systems by teacher, awarding bodies, students and general public DMW 2006 The Open University's Institute for Educational Technology

27. Visions: Personal • e-Portfolios exchanged maybe as microchips in business cards • All e-assessment except for practicals • e-Assessment seamlessly integrated into day to day practice • Peer e-assessment • Assessment for Learning • More use of diagnostics DMW 2006 The Open University's Institute for Educational Technology

28. Barriers: Superorganisational • Customer attitudes • Lack of public confidence • Negative stories in the media • Lack of integration of policies • Key antenna of quality, accessibility, reliability and security are evaluated with pilot activities DMW 2006 The Open University's Institute for Educational Technology

29. Barriers: Organisational • Lack funding to encourage pilot and R & D • Lack of recognition in RAE • Not engaging with in proven technology • Lack infrastructure to pilot • Lack of item banks DMW 2006 The Open University's Institute for Educational Technology

30. Barriers: Personal • Staff attitude • Availability of resources • Lack of IT skills in a pilot project’s target student population DMW 2006 The Open University's Institute for Educational Technology

31. Barriers to adoption • Reformation of methods • Training and development • Resources • Technical • Customer attitude • Staff attitude • Learner attitude DMW 2006 The Open University's Institute for Educational Technology

32. Recommendations: Superorganisational • Clear policy directives • Government agencies to support directions • Standard system not acceptable to awarding bodies • Standards for systems spelt out by QCA • Initiatives to share expertise • Recognition of training • Encourage open source licensing models • Co-ordinate dissemination at BECTA/JISC levels DMW 2006 The Open University's Institute for Educational Technology

33. Recommendations: Organisational • Tool development • Examine role of VLEs on progress • Follow SQA’s lead COLA and SOLAR • Find self and peer assessments DMW 2006 The Open University's Institute for Educational Technology

34. Recommendations: Personal • Training • Recognition of effort/time • Assist champions • Fund studies of changes to job allocation and staff culture • Appoint mentors DMW 2006 The Open University's Institute for Educational Technology

35. DMW 2006 The Open University's Institute for Educational Technology

36. Summary • Policy pressure main driver in England and Wales • Affects FE more than HE • More reliance than maybe anticipated on the commercial sector • More use of formative e-Assessment • More research funding into text recognition becoming acknowledged • More interactive assessment rather than MCQs needed DMW 2006 The Open University's Institute for Educational Technology

37. Moving forward • What’s missing? • Vision • Barriers • Recommendations DMW 2006 The Open University's Institute for Educational Technology