BENEFITS AND OBSTACLES: FACTORS AFFECTING THE UPTAKE OF CAA IN UNDERGRADUATE COURSES

1. BENEFITS AND OBSTACLES: FACTORS AFFECTING THE UPTAKE OF CAA IN UNDERGRADUATE COURSES John Dermo, Pathfinder Project Team, University of Bradford j.dermo@bradford.ac.uk 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007

2. First, a plea for your help…. “The idea being that authors use the conference to air their work and receive feedback to enable judgements to be taken on progress to date and how things may be taken forward in future.” (extract from Submissions: 11th CAA Conference) http://www.caaconference.com/submissions/index.asp

3. Background to the Research Project University of Bradford • e-Strategy • VLE = Blackboard ™ • CAA = QM Perception™

4. Developments in CAA @ UoB so far include: • Deciding on Questionmark™ Perception™ as our supported enterprise level software for online summative assessment • Encouraging and supporting its use in formative assessment and feedback • Centralising the administrative support for all summative assessments in our Examinations Office • Implementing Questionmark™ Perception™ version 4.3 with a server configuration to ensure security and reliability

5. Pathfinder Project @ UoB Investment in • CAA • Enhancing FYE March 2007-April 2008

6. Pathfinder Project: CAA Aims and Objectives • to embed support processes for e-assessment • to develop the administrative and support systems to ensure reliable and secure large-scale implementation of CAA with first year students so that we can subsequently roll this out to all students

7. Pathfinder Project: CAA Deliverables A system to support the effective implementation of formative and summative computer-aided assessment across the university, consisting specifically of: • baseline study and needs analysis report on use of CAA at UoB • research data (quantitative and qualitative) on use of CAA in undergraduate courses • official guidelines for CAA: admin and support functions to ensure resilient and manageable online assessment routes • pedagogical guidelines (“best practice”) document for recommended use of software • training sessions for instructors (Sept/Jan/May) • induction and training sessions for invigilators • in-house training materials and web-based resources for use of software • case studies of institutional implementation of CAA • banks of online assessment materials

8. Aims of my research project • gather data on attitudes towards computer-assisted assessment • to identify drivers and obstacles to the uptake of CAA • test various widely-held assumptions about attitudes towards e-learning • inform Pathfinder project • anticipate risks

9. Methodology • descriptive • survey opinions of comprehensive range of interested stakeholders: • students, academic staff, administrators, technical and support staff, management and financial departments. • primarily a qualitative approach

10. Phases of research • Informal interviews, desktop research to identify key areas of interest • Online questionnaires • Follow-up interviews to investigate issues arising

11. Challenges: methodology • Quantitative vs Qualitative • Positivist vs Anti-positivist • Survey vs Case study How can we draw generalisable conclusions based on individual personal opinions?

12. Challenges: methods • Mode of delivery (online vs paper) • Item design (open vs closed) • Subject groups and sampling • Piloting • Assuring confidentiality • How to survey opinion on something never experienced?

13. Important Note: Spend time at the outset to make sure that the research tools are reliable and valid before moving to data collection. All too often researchers rush into the data collection stage, only to find that the data they are collecting are useless and any findings based on it equally so.

14. Initial findings and looking ahead • research still at initial phase • key areas of interest being put to subjects to inform the main survey questions to come • reviewing the literature • It is hoped that feedback from conference delegates will be able to feed into this process!!

15. Drivers

16. Obstacles 1

17. Obstacles 2

18. Issues to consider • Does the cumulative effect of the barriers outweigh that of the drivers? • How to interpret the data in a meaningful way? • To what degree can e-assessment add value to the learning experience? • Do some factors work as drivers under some circumstances but as obstacles in others?

19. Closing comment… The researcher is keen to involve the input of other experienced practitioners and researchers in the field of Computer Assisted Assessment by means of this conference, and hopes to work this short paper up to a full paper submission for the next event in 2008.

20. References Cohen, L., Manion, L., Morrison, K. (2003) Research Methods in Education. London: RoutledgeFalmer. Conole, G. (2003) Understanding Enthusiasm and Implementation: e-Learning Research Questions and Methodological Issues in JK. Seale (ed) Learning Technology in Transition: from Individual Enthusiasm to Institutional Implementation, Abingdon: Swets and Zeitlinger, pp. 129-146. Dillman, DA. (2007) Mail and Internet Surveys: the Tailored Design Method. Hoboken, New Jersey: John Wiley & Sons. Frankfort-Nachmias, C. and Nachmias, D. (1996) Research Methods in the Social Sciences. London: Edward Arnold. Oppenheim, AN. (2000) Questionnaire Design: Interviewing and Attitude Measurement. London: Continuum International. Verma, GK. and Mallick, K. (1999) Researching Education: Perspectives and Techniques. London: Falmer Press.

21. Thank you Contact: John Dermo Pathfinder Project, Learner Support Services, University of Bradford Email : j.dermo@bradford.ac.uk