Evaluating your e-learning course

1. Evaluating your e-learning course LTU Workshop11 March 2008

2. Common practice • End-of-course student questionnaire • Standard version with last-minute adaptations: • changes to how the course elements are named • including some additional items Usually with similar problems… • low return rates • no follow up returns, and sometimes do not analyse results • poor quality responses • limited enhancement function

3. Research shows… Evaluation of e-learning courses mainly affected by: distributed configuration of teams and distance of students • Strategies for evaluating courses are problematic even at compliance level • Sense of evolution – looking for the right strategy • Teams worry about response rates but don’t use data available • Undefined responsibilities • Staff’s lack of skills, confidence & motivation

4. Common problems • Evaluation is not planned (in advance) as part of the overall course design • Feedback is gathered only from students • Feedback is gathered only at the end of the course/module • Feedback is gathered only through traditional methods • Feedback requested do not consider all e-learning aspects

5. Recommendations& Strategies

6. Collect feedback from tutors, administrators and technical support • Usual context: team meetings affected by distribution of staff (part time or at a distance). • Swinglehurst (2006) approach – tips for e-learning course teams: • Formally organise frequent staff meetings, where tutors, administrators and support staff meet to analyse course. These meetings could be face-to-face and/or online. • Define an agenda for each meeting, and rotate the member of staff responsible for this agenda, to cover key issues related to the different aspects of the course (e.g. student support, academic feedback, encouraging discussions) • Limit the recording of the discussion at these meetings to a minimum, briefly documenting the topics covered and the decisions made, and disseminate them. • Questionnaire to staff

7. Collect feedback during the run of the course • Daly et al. (2006) approach - offers a simple and effective mechanism for the collection of student feedback during the run of a course: • embedding evaluation activities, encouraging students to think about how the course design/materials/activities have support their learning. • Main benefits: to identify difficulties when they students are experiencing them, and the opportunity to explore students’ experience. • Online learning diaries - running throughout the course, in which students are encouraged through brief questions to post their thoughts regarding their learning process and how the course has supported them.

8. Collect feedback at the end of the course/module • Questionnaires • Questionnaire/question features: language, questions types • Mode of application: • Online (using an online survey tool) • By e-mail • Paper-based • Timing at which feedback is collected • Responsibility for collection and analysis • Focus groups

9. Use alternative strategies • Computer logs that the particular e-learning system in use provides (e.g. Course Statistics in the Blackboard VLE). • Monitoring basic statistics such as last login date, number of messages sent by users, areas of content and discussion boards/forums visited by users. • Statistics are not an indications of the quality of the student/tutor participation or of a satisfactory online experience – but useful tool for monitoring online ‘presence’, to obtain an overall picture of the ongoing activity, as well as to detect problems

10. Include aspects relevant to the use of technology • Wide range of aspects to include, and this list would vary depending on context and on the objectives and audience for the evaluation • Main issues: • quality, usefulness and frequency of use of course components (online activities, resources, face-to-face events, readings, online discussions/seminars, tutor support, technical support, etc.) • how well online activities run (timing, frequency, sequence, instructions, interactions, feedback, time on task, etc.) • e-learning experience (workload, involvement, online participation facilitators and restrictions, etc.) • role of tutors (engagement, feedback, support, etc.) • Warwick University and Bristol University’s guidelines IHEP’sQuality on the Line report (Phipps and Merisotis, 2000)UCL’sQuality Framework (Greenhalgh, 2001)