Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert

1. Aligning the Ducks: Sustaining the quality of CAA in a Period of Growing Demand Glenis Lambert LTEU, Canterbury Christ Church University Glenis Lambert LTEU June 2009

2. Glenis Lambert LTEU June 2009

3. Outline The history of online summative tests at CCCU using short-answer questions delivered online A reflection, 2005 – 2009 What do we want from CAA? How do we match up to this today? The way forward Glenis Lambert LTEU June 2009

4. Background: Canterbury Christ Church University Faculties of Health and Social Care, Education, Undergraduate Programmes +++ Large numbers of students on placement Diverse student body Growing numbers of collaborative programmes Multi-location campus Growing student numbers Central Learning and Teaching Enhancement Unit with Learning Technologists for each Faculty Glenis Lambert LTEU June 2009

5. 2005 What we did Bought QuestionMark/Perception Mapped with existing University policies and managed conflicts Using BS ISO/IEC23988:2007 Wrote a policy document Set up staff procedures and guidance with full documentation for staff What happened in the pilot phase Over the academic year 8 end of course exams in 18 sessions were successfully delivered, totalling 660 individual examinations. 5 of these were organised and delivered by the departments, three by the examinations office. 2005 2006 2007 2008 2009 Glenis Lambert LTEU June 2009

6. 2006 What we did • Slight increase in number of tests • Consolidated procedures What went wrong • Lack of staff confidence in online environment led to high support demand • Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student • Human error! • What we Learned • Stakeholder confidence in support and procedures help develop practice • Buy-in at all levels is essential • Clear fall-back strategies are essential • BUT: • Exit surveys revealed student satisfaction with online examinations • All of the fall-back procedures worked well • The systems/software performed well 2005 2006 2007 2008 2009 Glenis Lambert LTEU June 2009

7. 2007 What we did • Delivered over 5,000 summative tests and a • few other surveys. • Support load on LTEU began to be unsustainable. • What went wrong • Staff still not engaging with processes and procedures • Infrastructure made process expensive • Some institutional procedures did not fit the CAA mode of delivery, resulting in complications for the student • Human error! • “Last minute” culture led to increased support demand • Infrastructure began to creak! 2005 2006 2007 2008 2009 Glenis Lambert LTEU June 2009

8. 2008 What we did to try to manage the support load • Put an embargo on all new tests and set out to reinforce our policies by working with stakeholders to devolve some of the activities carried out by LTEU • Bought the Blackboard Bridge to reduce the administrative load • Encouraged more academics to do “open” assessments which reduced the infrastructure and invigilator load, but not the support load, although this changed its emphasis. (Bertolo and Lambert, 2007) • What we learned • Our ambition to control the testing environment was probably never going to happen • It was going to take a long time to get all stakeholders on board • There needed to be a central agency to co-ordinate all aspects of the process • Support time for CAA was always going to be high • Infrastructure/development costs were an ongoing commitment • We had yet to muster a persuasive argument to get total institutional buy-in 2005 2006 2007 2008 2009 Glenis Lambert LTEU June 2009

9. 2008 What went well • We have rarely had to resort to “fall-back” positions, e.g. paper • Our quality procedures have been robust and applied throughout • Our planning procedures have raised stakeholder confidence • Documentation produced for staff has been welcomed resulted in clear definitions of responsibilities • Staff development has led to improvements in all areas 2005 2006 2007 2008 2009 Glenis Lambert LTEU June 2009

10. 2009 • What we learned • We needed to think carefully about what we were doing and whether we should continue to do it. 2005 2006 2007 2008 2009 Glenis Lambert LTEU June 2009

11. What do we require from CAA? Report on eAssessment Quality (REAQ) Overview There is wide consensus that e-assessment promises many potential benefits to the JISC community, ranging from lowered costs, higher productivity, and faster feedback, through to assessments which are more accurate, more detailed, and more robust under critical scrutiny and audit. There is equally wide concern that these benefits seem relatively slow to materialise, and may be negated by poorer quality assessment which fails to reliably provide acceptably valid measures of actual student achievement and capability, particularly at the higher levels of Bloom’s taxonomy of the cognitive domain. The issues confronting staff concerned with the quality management of e-assessment suggest a pressing need for evidence, for information, and for guidance. http://www.jisc.ac.uk/whatwedo/projects/reaq.aspx Glenis Lambert LTEU June 2009

12. What do we require from CAA – the Institution’s view? Assessed tasks should attract sufficient student time and effort Tasks should engage the student in productive learning activity Students' effort should be spread as evenly as possible Orientation to the task should help students to perceive the demands of an assessment task Communication through guidance provided should offer clarity on expectations Sufficient and timely feedback should be provided Feedback should focus on learning not on the marks or the student Feedback should be closely linked to assessment criteria Feedback should be given at a level accessible to the student's sophistication Students should be required to receive and respond to feedback http://www.canterbury.ac.uk/support/learning-teaching-enhancement-unit/assessment/assessment-handbook/pages/section1/11-3-strategies.asp Glenis Lambert LTEU June 2009

13. Can CAA ever be “fit for our purposes?” Develops students’ abilities to evaluate own progress, direct own learning Emphasises authentic and complex assessment tasks Uses high-stakes summative assessment rigorously but sparingly Offers “low-stakes” confidence building opportunities and practice Is rich in formal comment, (e.g. tutor comment, self-reflective logs) Is rich in informal feedback (e.g. peer review, collaborative project work) Above all, it should enhance learning. Glenis Lambert LTEU June 2009

14. Quality Assurance Procedures ‘Openness to challenge is a critical cultural necessity for good risk management and compliance – it is in fact more important than any framework or set of processes’. Paul Moore, Evidence to Treasury Select Committee (February 2009) “Over the last twenty years, the higher education sector has been subject to structural and cultural changes associated with a focus on accountability and value for money which have profoundly affected academic life…..  In this new reality the term ‘quality’ became synonymous with data collection, performance scrutiny and a massive increase in bureaucracy. ” George Gordon and Catherine Owen, Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners   Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland (2009) Glenis Lambert LTEU June 2009

15. What are the risks? Risks involved in the continuing growth of CAA: High costs of quality-assurance not met by institution High costs of reliable delivery not met by institution Concentration on “box ticking” accountability does not lead to enhanced learning Risks of NOT using or encouraging the growth of CAA: Opportunity for enhancing learning and feedback lost Students’ expectations of HE not met (Dermo 2009) Glenis Lambert LTEU June 2009

16. What we want from CAA Embedding Embedding • In addition: • That it is used to provide speedy feedback to students • That it is used as a means of enhancing learning Strategy Strategy Policy Policy Robust Robust Reliable Reliable Fit for Purpose Fit for purpose Glenis Lambert LTEU June 2009

17. Where we are at present Where people have used CAA it has become embedded in their practice, but summative use isn’t leading to formative use. Embedding There is a growth in the strategic use of CAA, but largely to reduce marking load. Strategy Policy documents are now in existence, but they need reviewing in the light of support demand. Policy We are confident that our policies and procedures do result in adequately assured assessments where they are applied, but the infrastructure and support services are sometimes more fragile Robust The software and systems have proved to be reliable. Infrastructure and the application of policy and procedures less so. Reliable “Quick Burn” implementation (Warburton 2009) leading to some questionable practice Feedback not built in to tests, little formative testing Fit for purpose Glenis Lambert LTEU June 2009

18. Conclusions The support and maintenance costs of quality-assuring CAA are high and ongoing It is easy to quality-assure CAA if it is only seen as a method of providing accountability for assessment practices and procedures If the major requirement of CAA is to enhance learning, it can be cost-effective, when “opportunity costs” are factored in (Ridgway et al. 2004). To achieve the benefits of CAA for learning enhancement staff development has to be the central strategy for assuring quality. A “student satisfaction” ethos needs to be central to mitigating failures Glenis Lambert LTEU June 2009

19. Forward plans • Raising awareness through Enhancement Theme on Assessment and Feedback • DEBUT staff development project • Work with staff on planning of new courses via FQOs • Development of e-learning strategy • Benchmarking e-learning, including use of CAA • Inclusion of policy and procedure documents in staff development pack Embedding Strategy Policy Continue to work with all stakeholders to emphasise the importance of team structures Staff development for all stakeholders including Examinations Office, User Technology etc. Robust In the absence of more resources, designate a number of rooms as testing labs to make infrastructure support more efficient Make sure all stakeholders know about fall-back procedures Reliable Maintain central consultancy role in LTEU for construction of questions and tests DEBUT staff development project Fit for purpose Glenis Lambert LTEU June 2009

20. Glenis Lambert LTEU June 2009

21. References Assessment Handbook, Canterbury Christ Church University http://www.canterbury.ac.uk/support/learning-teaching-enhancement-unit/assessment/assessment-handbook/pages/section1/11-3-strategies.asp Bertolo, E & Lambert, G: 2007 Implementing CAA in Chemistry: A Case Study Procedings of 11th CAA Conference, Loughborough University 2007 Dermo, J. (2008) “Implementing Online Assessment: Finding the Right Path for an HE Institution” in Ladwa, A (ed) E-Learning in HE available online at http://www.rsc-yh.ac.uk/Documents/HEbooklet2_000.pdf (last accessed June 2009) Gordon, G & Owen, C, 2009,Cultures of Quality Enhancement: a short overview of the literature for higher education policy makers and practitioners Scottish Higher Education Enhancement Committee (SHEEC) and QAA Scotland Parsons, R, (2004) Ensuring Quality and Efficiency with Online Assessments JISC http://www.enhancementthemes.ac.uk/documents/events/20040416/Parsonspaperrevised.doc. Report on eAssessment Quality (REAQ), JISC 2009 http://www.jisc.ac.uk/whatwedo/projects/reaq.aspx Accessed June 2009-07-02 Warburton 2009, Quick win or slow burn: modelling UK HE CAA uptake Assessment & Evaluation in Higher Education, 1469-297X, Volume 34, Issue 3, Pages 257 – 272 Glenis Lambert LTEU June 2009

22. What we want from questions and tests That online assessments are viewed as routine, in the same way as presentations, paper tests etc. Embedding That short, formative online tests are a routine part of assessment strategies Strategy That policies and procedures are routine elements of assessment procedure Policy That all questions and tests are valid and peer-reviewed Robust That staff can rely on support and staff development in the preparation of assessments Reliable That all elements are “fit for our purposes” Fit for purpose Glenis Lambert LTEU June 2009

23. The question question: a digression. Stage 1: This is a duck: True or False? Stage 2: This bird is: A coot c) A duck A swan e) A diver Stage 3: Identify this common duck and match it with its species Mallard i) Marmaronetta Pochard ii) Netta Teal iii) Aythya Glenis Lambert LTEU June 2009

24. Barriers to improving question quality Many academic staff operate in a text-based environment and find the black and white nature of computer-marked questions challenging Learning technologists do not have detailed subject knowledge Staff development to enable the development of question-setting skills takes time. Glenis Lambert LTEU June 2009