Evaluating the Impact of a VLE on Learning and Teaching?

1. Evaluating the Impact of a VLE on Learning and Teaching? Dr Barbara Newland Learning Design Studio Academic Services Bournemouth University http://www.bournemouth.ac.uk/lds/ bnewland@bournemouth.ac.uk

2. Overview • Aims of evaluation • Durham context • Methodology • Results • Conclusions

3. Aims of evaluation • International widespread adoption of VLEs • Longitudinal study of impact of a VLE on learning and teaching • Qualitative and quantitative methods

4. University of Durham • 670 academic and 280 academic-related staff • 9400 undergraduates and 2400 postgraduates • Founded in 1832 • Split campus • Research led • MIS – outsourced to Unisys • Learning Technologies Team (LTT)

5. Images of Durham

6. Learning Technologies Team • aims to encourage, support and develop elearning throughout the University by: • providing advice, information and training • supporting the University's learning environment - duo (Durham University Online) • developing interactive materials • 4 members of staff – Juliette Pavey, Kate Boardman, Malcolm Murray, Victoria Boyd

7. Duo • Durham University Online • http://duo.dur.ac.uk

8. Uptake of duo

9. Implementation • Varied patterns of departmental implementation • All courses • All first year courses • Individual academics • Administration courses • Duo is now embedded into learning and teaching

10. Methodology for duo surveys • Quantitative and qualitative • Online surveys • Analyse factors such as gender, year of study and faculty • 3 faculties – Arts and Humanities, Science, Social Science and Health • Relate to C&IT skills in learning and teaching • Data collected from students and staff in all faculties and both campuses

11. Response rate • All students and staff are in duo so hard to know the response rate in relation to active users which has increased over the years • Staff April 01 (n = 91) May 02 (n = 153) May 03 (n = 283, c. 40%) • Students May 01 (n = 793) May 02 (n = 2435) May 03 (n = 3668, c. 37%)

12. Survey questions • Similar questions for staff and students in all 3 years to enable comparisons : • between years • between staff and students • Software • ease of use • levels of confidence • features used

13. Survey questions • Frequency of use and access during term time and vacations • Learning and teaching • contribution • effect

14. Student responses – support, frequency and access • Help documents • 19%, 15%, 10% • Small and decreasing use • Accessed duo at least once a week • 70%, 71%, 71% • Consistent pattern but increasing numbers over time • Accessed duo out of term time • 60%, 61%, 64% • Consistent pattern but increasing numbers over time

15. Students: Overall, how do you rate the contribution duo has made to your learning?

16. Staff: Overall, how do you rate the effectiveness of duo as a learning resource?

17. Staff: Has duo affected your approach to teaching?

18. Students: If duo is not currently being used for all your modules, would you recommend that it should be?

19. Staff use of duo features - content • 62%, 49% staff have used more features • 79%, 76% staff have developed content since the previous year • Course information – 74%, 84%, 92% • Course documents – 77%, 89%, 95% • External links – 42%, 56%, 55% • Bibliography – 35%, 53%, 43% (plus 17%, 21%, 21% link to OPAC) • Staff information – 49%, 63%, 72%

20. Staff use of duo features - communication • Group work –25%, 23%, 26% • Email – 72%, 77%, 80% • Announcements – 75%, 84%, 86% • Discussion board–16%, 32%, 31% • Chat – 10%, 4%, 6%

21. Staff use of duo features - assessment • Assignments – 53%, 63%, 67% • Quizzes – 11%, 20%, 26% • Surveys - 15%, 13%, 21% • Digital drop box – 10%, 6%, 15%

22. Reflect current ways of teaching • Context – relevant, flexible (start from where they are at) • Collis also found at the University of Twente that “instructors are most likely to begin by choosing aspects of a system that reflect their current ways of teaching, and then gradually move to new instructional approaches and new features.” (Collis and Messing, 2001)

23. Lecture notes - staff • Is it good practice to put lecture notes online? • Depends on what the lecturer does with them • 2/3 put lecture summaries, handouts, full notes and/or Powerpoint slides after the lecture

24. Lecture notes - students • Depends how students use them • 73% listen more • 81% clarify information • 72% improve lectures • 93% study for exams • 2/3 (64%, 64%) students note-taking had stayed the same but for ¼ (27%, 26%) it had decreased • Only 1%, 1% stated no longer attended lectures

25. Gender • No statistical significance between gender and • Staff with effectiveness as learning resource • Students with contribution to learning • Both with levels of confidence • Higher percentage of females think duo helps them to manage their learning

26. Staff by faculty – skills • More Science faculty found it very easy/easy to use compared to Arts • Some Arts staff took longer to gain confidence (6-10 visits) • Fewer Science staff chose to have training • Reflects the levels of staff confidence in C&IT skills found in Audit June 2000

27. Staff by faculty – teaching • 2002 - no significant difference • 2003 • Basic approach is changing • Science and social science 12% compared to 3% arts • Basic approach has not changed but duo helps me do certain things better • Science 79%, social science 76% compared to 98% arts

28. Students by faculty • Contribution that duo has made to their learning overall • 2002 - no significant difference • 2003 – higher percentage of science (69%) and social science (52%) rate “good” or “excellent” compared to 42% arts students • Higher percentage of social science students think duo helps them to manage their learning “greatly” or “quite a lot” compared to science students

29. Students by year group • Contribution that duo has made to their learning overall • 2001 and 2002 - 2nd and 3rd year students rates more higher than 1st years (1st years have always had duo) • 2003 – 1st year students rate slightly higher than 2nd and 3rd years

30. C&IT skills survey of new undergraduates – 2002 • n = 2656 • 82% of students perceive themselves to be confident/very confident computer users • 60% have brought a computer with them to University • 69% have been using a computer for 5+ years (including 21% for over 10 years) and only 2% have been using a computer for <2 years

31. C&IT skills survey of new undergraduates - 2002 • 79% perceive themselves to have good/advanced email skills • 65% perceive themselves to have good/advanced web browser skills • 70% stated a preference for a combination of web and paper based for learning support materials

32. C&IT skills survey of new undergraduates - 2002 • 95% own a mobile phone • 5% have a PDA • 38% take part in online discussions • 59% take part in online chat • 49% have a qualification in IT • 54% regularly switch between several applications when using a computer

33. C&IT skills survey of academic staff - 2000 • n = 576 (53%) • 90% perceive themselves to have good/advanced email skills • 67% perceive themselves to have good/advanced web browser skills • only 17% of staff had taken part in online discussions • only 12% of staff had taken part in online chat

34. Catalyst • Introduction encouraged continual reflection on teaching • Postgraduate Certificate in HE compulsory for new academics from Jan 2001 • Academics working together within courses

35. Factors influencing enhancement • How used • How integrated into curriculum • Quality of the parts eg content, assessment, communication (earlier research) • Enhance learning if the parts enhance learning • Flexibility allows lecturers to choose parts

36. Is the whole greater than the sum of its parts? • But is a VLE more than the pieces put together?

37. Evaluating the impact of a VLE on learning and teaching • Incredible uptake of duo which has become embedded into the University • Staff and students believe duo has enhanced teaching and learning • Academics are gradually using more features so this may result in more interactive and innovative teaching • Students expectations are increasing as the basic use of a VLE is assumed • Further analysis is being undertaken

38. References • Collis, B. and Messing, J. (2001) Usage, attitudes and workload implications for a web-based learning environment, Association of Learning Technology Journal 9 (1) • Collis, B. and Moonen, J. (2001) Flexible Learning in a Digital World, London, Kogan Page • Laurillard, D. (1993). Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. London: Routledge. • Richardson, J., & Turner, A. (2000). A large-scale ‘local’ evaluation of students’ learning experiences using virtual learning environments. Educational Technology and Society, 3 (4). • Salmon, G. (2000). E-moderating: the Key to Teaching and Learning Online. London: Kogan Page.