JISC/NSF DIALOG-PLUS

1. JISC/NSF DIALOG-PLUS Bi-annual Meeting, PSU 17th-19th September, 2005 The Challenges of On-line Learning in the Undergraduate Curriculum: Delivering a Module on Earth Observation Louise Mackay j.l.mackay@leeds.ac.uk Today’s report: (i) Summary of Earth Observation nuggets developed at Leeds; (ii) Introducing nuggets to the undergraduate curriculum – using EO nuggets as course lectures; (iii) Delivery of a complete EO course; (iv) Learning and teaching experience of the course/delivery; (v) Issues risen and lessons learnt!

2. Summary of Earth Observation material at Leeds… The Earth observation nuggets at Leeds have been developed (during Spring 2004) from the EO component of the Geography module: Geog2750 Earth Observation & GIS of the Physical Environment The Earth observation component delivered first semester of a full 2 semester module - GIS covered in semester 2 The EO semester – an 11 week course: a 1 hour lecture per week and a two hour practical session each week. Module convenor: Dr Steve Carver; Course tutor: Dr Stuart Barr (now at University of Newcastle) Lecture material - physical principles of Earth observation Practical material - principles of processing optical and microwave environmental image data (lab based only) The course in its original guise – 2003/2004 – taught face-to-face

3. Summary of Earth observation material at Leeds… The Earth observation nuggets at Leeds have been developed from the Earth Observation lecture material (Semester 1) of the Geog2750 course. Nugget material – series of 10 lecture style nuggets plus self-assessment MCQ test based on the course materials Level – 2nd year Geography undergraduates Learning objective – Introducing the physical principles of Earth observation and its role in physical geography applications Deposited – On the D+ Swiki site and the Leeds Nathan Bodington VLE What we use at Leeds: (i) Landsat TM of the UK (ii) ERS SAR of Svalbard glaciers (iii) Lidar and DSM’s of UK study sites Software available: Erdas Imagine and Esri IDRISI

4. An opportunity to use D+ nuggets in the curriculum… Remember – EO one of the core Geography sub-areas of the D+ project and the material content deemed useful for Leeds and collaborating partners How do we use developed material to enhance undergraduate teaching? (or how do we maintain valuable material in the undergraduate curriculum?) Having created D+ nuggets from an existing course in Spring 2004 and in the light of staff changes… Take nuggets developed from the original EO course - deliver the course in a different format: predominantly on-line and distance-taught

5. An opportunity to use D+ nuggets in the curriculum… • The provision of the course in this format, by using all developed EO nuggets fulfilled three main objectives: • Delivery of knowledge and skills to the students on Earth Observation within a EO & GIS module; • Experimentation with a mixture of face-to-face and distance tutoring on an UG module; • Preservation of learning materials from year to year. • The “new” Geog2750: the Geog2750 course in its new guise was taught to 56 Level 2 students at Leeds during semester 1 of academic year 2004/2005 using EO nuggets as on-line lectures, supported by distance-tutoring and 8 on-campus practical labs.

6. The “look and feel” of Leeds EO nuggets used as on-line lectures… • Each of the 10 nuggets/lectures is designed to take 1-2 hours to complete. • Each lecture is structured with an introduction, aims, signposted sectioned content with specific Q&A points, links to relevant reading/on-line resources, summary and linkage to the lecture that follows. • Each lecture also comes with a self-assessment MCQ test.

7. Example on-line lecture and MCQ material The lectures have a distinct “tone” to that of face-to-face lectures in order for students to work unaided. The on-line versions have been adapted to add more to the learning experience by using graphical examples and external links to image sources and tutorials; with the suggested readings this consolidates the introduced topics and directs additional self-study. MCQ example Lecture examples

8. Delivering the on-line lectures through the Leeds VLE (Autumn 2004) The on-line lectures (plus MCQs) were delivered using the Leeds Bodington Common VLE. Course “room” resources: Course introduction/advice; Practical material; Discussion and FAQ room; Lecture MCQ test; Additional Resources; Miscellaneous Resources The lectures were supplemented by email tutoring contact with the new course tutor (LM) and by maintaining the practical sessions at Leeds during the autumn 2004 semester (taught on-campus and with demonstrator assistance).

9. Delivery and assessment of the “new” Geog2750 course • Delivering material and support during the course: • Weekly lectures and MCQ’s added to the VLE room each week • Practical material added to the VLE room prior to on-campus practical classes • Practical advice to demonstrators prior to on-campus practical classes from the tutor • Directed queries to form the basis of tutorial queries - for publication to the class discussion room. • Weekly supportive emails from the tutor to the class introducing the week’s material • Not to mention - An open ear at all times for student and demonstrator queries! • Assessment of the course: • 5 assessed practical assignments plus an end of semester examination (held January 2005) • The course was successfully delivered to 56 students in Autumn 2004.

10. An opportunity to evaluate Geog2750 A tutor visit during the final teaching week – final lecture (of the series), small group tutorial sessions and workshops and one-on-one student meetings To allay student fears as reported in the mid-term staff/student meeting During the tutor visit - tutorials and students successful! An opportunity to evaluate… Karen Fill visited Leeds to help evaluate the student and tutor experience.

11. A quick advertisement of the Leeds EO material!!! • How is the Leeds EO material useful? • Leeds EO material – useful at a generic level • The provision of EO nuggets acts as an additional tutorial for those students not enrolled on an EO course and provides understanding of how EO data/processing techniques can be applied in a wider context (e.g., land cover/land use mapping). • How can you use the EO nuggets? • The nugget series can be used in its entirety as a complete introduction to EO tutorial (e.g., for dissertation students) or as individual topic-specific nuggets (for “picking and choosing”). • Advantage for use elsewhere is in the nugget design: • sign posting and ease of navigation throughout the nugget; • can be “dipped” into for a specific need yet provides cumulative EO knowledge. • Where to get the material: the D+ swiki, the developer (LM) and soon to be on the JISC depository JORUM.

12. Evaluating the distance-based Geog2750 course • The course was evaluated in December 2004 (with the help of Karen Fill) during the final teaching week by means of: • student questionnaire, group discussion and tutor interview; • and after the end of semester examination (February 2005) by: • statistical analysis of inter-year marks (assessments and examination) • What did we learn from this evaluation?...

13. Evaluating the distance-based Geog2750 course - Student questionnaires Thirty-five (63%) of the 56 students taking the course attended the last lecture and completed the questionnaire. None had studied by distance learning before. For the questionnaire the students were given two tables containing statements regarding the content and delivery of the course and asked to think about their use of the lecture and score each statement according to their level of agreement (0 – no, 1 – somewhat, 2 - yes, and N/A). A mean agreement score was calculated for each statement. Example of questionnaire statements on style and content

14. Evaluating the distance-based Geog2750 course - Student questionnaires • The agreement scores that occur: • Best scores - description, interface and tools (mean agreement score 1.8-2.0) (i.e. the physical presentation and style of the lecture); • Worst scores - motivation, support and learning styles (with low mean agreement score ≤1.5) (i.e. the distance-based delivery of the course). • The most strongly agreed statement was that of statement 1: • “there was a full description of each lecture, including learning objective”, 97% agreement, 3% somewhat agreement. • The least agreed statement was that of statement 22: • “I preferred this form of learning to that of timetabled face-to-face lectures”, 60% disagreement, 11% agreement, 29% somewhat agreement.

15. General questionnaire comments were as follows: • Surprisingly good way of learning – practical sessions were vital however in keeping me focused / on time. • The marking aspect of the distance learning was not a success. Due to the lack of a face-to-face lecture the marking of the practical assignments involved a great amount of inconsistency. Needs to be seriously addressed. • Need to have lectures at certain periods, even if only every few weeks. • As the vocabulary is quite technical it had not been easy because I am an Erasmus student. • Note - The lack of motivation and lack of contact due to the delivery being face-to-face were the types of comments that were repeated by many students across many statements.

16. Evaluating the distance-based Geog2750 course - Student group feedback • After the student questionnaire - seventeen of the students stayed for a discussion conducted on nominal group lines (Harvey, 1998) • 49% attending/30% of class • After some discussion (in groups): • Best aspects of the course: being able to work in one’s own time, reading lists, multiple choice quizzes and loads of information available. • Worst aspects of the course: didn’t meet many people, inconsistent marking of lab work, no face-to-face contact if problems arose, information overload. • Individual choices (16 students): • Majority Best: being able to work in their own time • Majority Worst: lack of face-to-face contact • Harvey, J. (Ed.) (1998) Evaluation Cookbook, pp. 44-45 (Edinburgh: Heriot-Watt University).

17. General discussion points: • Distance learning - questionable approach for difficult mathematical concepts. • They should have been told in advance. • Not what students came to university for (some outrage expressed). • Need to be able to ask questions. • Need contact to confirm you are right about things. • Email answers are needed within a day. • Distractions of the WWW. • Discussion board - better if anonymous/not always possible to explain oneself. • MCQs could stay up after lecture deadlines; helpful for revision.

18. Evaluating the distance-based Geog2750 course - Summarising student feedback • Student response to the Dialog-Plus nuggets was particularly positive with respect to: • - description of content and learning objectives • - inclusion of required tools • - accessibility of linked resources • appropriateness of the MCQs • Female students’ responses (with regards to tools, the learning style and preference of distance-based to face-to-face), were slightly (not strongly) more negative than those of male students. • The students gave particularly negative feedback about: • their own motivation and preferences for learning with respect to the on-line and distant tutoring aspects of the course delivery; • specifically the lack of face-to-face contact (both with the tutor and fellow students).

19. Evaluating the distance-based Geog2750 course - Tutor experience • So what about the tutor’s experience of the nuggets and their delivery… • Evaluation of the nugget material (by tutor interview): • How could the nuggets be improved? • - For transferability – location specific examples • - Further embedded exercises (e.g. Flash applications) • Evaluation of distance based learning (by tutor interview): • - Leeds VLE equipped to deliver nugget material • - Of additional resources MCQs were well used (VLE statistics) - students used them for revision after the course was taught • - Links to additional on-line academic tutorials – no access statistics • FAQ/discussion board room underused – students stated lack of anonymity/difficulty articulating problem • Overall distance-teaching aspects: • Main advantage - students problems and queries could be dealt with in “batches” • Main disadvantages - adversely the number of student queries was large and time consuming; lack of suitable administrative support for the course and (naturally) a feeling of isolation from the rest of the faculty.

20. Evaluating the distance-based Geog2750 course Analysis of class marks • Following examination (February 2005)… • Module marks tested (t-test) by gender and by year (2004 Vs 2005) • Statistical analysis of overall marks obtained by students in 2003/04 and 2004/05 revealed: • - no statistically significant differences year on year for all students; • - no statistically significant differences by gender within either year or across the years; • Analysis of the marks allocated for individual elements revealed that: • - students were awarded much higher marks on Practicals 1 and 2 in 2003/04; • - students were awarded much higher marks on Practical 4 in 2004/05. • Note - there is no statistical anomaly for female students for individual practical marks or overall module marks.

21. Issues arising… • What are some of the issues that have arisen and need to be addressed… • Student Expectation: • Students seemed to be disconcerted by the lack of on-campus support – undermined their own ability, i.e., • - Seemed to anticipate that all (non-tutor) means of support on campus was not available. • - Perceived that they hadn’t understood difficult technical topics • What could have given rise to this problem? • Students not familiar with distance-based learning • Lack of suitable advertising (positive reinforcement) of the course and its delivery style – especially for the inexperienced undergrad! • Lack of faculty support/communication with tutor • What about managing the tutor’s expectations? • Time and training – need to fully research and evaluate best-practice techniques for delivering material • Faculty support – lack of understanding for the need for a different type of support for a distance-based course

22. Issues arising… • Its not all bad… • Overarching advantage: • Opportunity to enhance student learning skills and experience – preparation for live-long learning • Other advantages: • Physical format of the materials – recorded and ready for use elsewhere • Mechanism to maintain integral element of the Geography undergraduate degree programme at Leeds • Distance-teaching provides a mechanism for an expert in the field to provide timely educational material and teaching • Distance-teaching - effective management of student queries

23. What lessons can we learn? • What can we do to make the experience better? • 1. Allay student fears with regards to delivery • How can we do this? - positive advertisement (e.g., student flexibility, one on one support with an expert via email) and at a time suitable for student’s module choices. • 2. Improve tutor support • How? – Faculty level training for on-line teaching;improve administrative/tutor communications – be clear on who does what specifically when distance-based. • 3. Allay student fears with regards to their ability • How? – provide early tutor visit/more on-campus “technical” support

24. Conclusions to be drawn… Students positive with regards to the content material that was developed for the course (nugget material). They liked the nuggets! Main student disadvantage: the current course format - lack of suitable support or face-to-face contact on campus Marks for the entire cohort were not adversely affected by the delivery format Tutoring on-line provided a means to manage class-wide queries, but distance-tutoring requires good faculty/administrative support To run again – advertise, support and provide on-campus backup (or visit)

25. My final comment on the “Geog2750 experience”… • Although a distance-based learning course appears to the inexperienced undergraduate student to be a “step too far” – not necessarily the case if: • - The teaching content is good quality • - Suitable understanding of the benefits of distance-based learning • - Prepared to engage with the support given… • Then they need not be disadvantaged by the experience!

26. Report summary… • Report Summary: • Roll out of 10 EO nuggets to Geography undergraduates in autumn 2004 through the Leeds VLE; taught on-line as the main teaching element of Geog2750, supported with distance-tutoring and on-campus practicals. • Evaluation of the teaching and learning experience of using developed nuggets with a distance-based delivery style. • Positive and negative learning and teaching experiences – negative responses mostly due to lack of undergraduate experience and lack of face-to-face contact. • Positive responses to the nugget content and style!! • Student marks not adversely affected and delivery style provides a useful mechanism to maintain a Geography topic in the undergraduate curriculum. • Dissemination:Mackay, Rees and Fill, JGHE 2005. • Open to questions/comments!