Evaluating Distance Learning: Feedback from “Distressed” Students

1. Evaluating Distance Learning: Feedback from “Distressed” Students Christopher Essex Kursat Cagiltay Indiana University AECT National ConventionDenver, ColoradoOctober 27, 2000

2. Our Background in Online DE • Essex has worked for the Indiana University SOE Distance Education Program since 1994 • Essex has taught online since Summer 1998 • Cagiltay taught Turkish students online from USA

3. Outline • Definition of “Distress” • Study Background • Course and Student Profile • Method • Findings: Positive and Negative Feedback • Recommendations • Limitations

4. Online Distance Education Student “Distress” Key References • Hara, N. and R. Kling (in press). Students' distress with a web-based distance education course. Information, Communication and Society. (Earlier draft is on-line:Available: http://www.slis.indiana.edu/CSI/ wp99_01.html) • Mendels, P. (Sept. 22, 1999). Study Finds Problems With Web Class. New York Times Online Edition. (On-line). Available: http://www.nytimes.com/library/tech/99/09/cyber/education/22education.html • Gale, C. (2000, January). Online learning: A student perspective. Syllabus. pp. 52-53

5. Online Distance Education Student “Distress” • Distress: “situations that the students...find particularly troublesome.” • Social Isolation • Overwhelming Email Communication • Lack of Instructor Feedback • Technical Problems • Ambiguous Instructions • (Hara and Kling)

6. Study Background • The Course • Focused on the evaluation of Internet resources for use by K-12 educators and students • Course activities • Required students to navigate through various websites, the course website along with a number of others, send and respond to email messages, and use a web-based conferencing tool for class discussion purposes • Instructor’s role/pedagogy • Facilitator, guided students through PBL-based activities and moderated class discussion

7. Study Background • Our role as evaluators: • The researchers conducted the study from a third-party perspective, observing impartially and not involved in making any changes to the program based on our evaluation.

8. Course Profile • 100% online course, utilizing course website, email and asynchronous web-based conferencing • Master’s-level, three-credit hour course • Designed and taught by advanced doctoral student • Part of an online Master’s degree program

9. Student Profile • Of 11 registered students, 9 responded to at least one of various survey instruments. Seven students filled out all the instruments • 100% graduate-level students • Students were located in Indiana, Hawaii, Iowa, Tennessee, North Dakota, and Sweden • 4 in on-campus degree program • 1 in online master’s degree program • 10 enrolled for educational and business-related reasons--Only 1 registered for personal reasons

10. Methods • Interviewed instructor • Reviewed course content/activities • Developed questions based on Kirkpatrick’s Four Levels of Evaluation • Developed online survey instruments • Integrated surveys into course website

11. Focus of Kirkpatrick Levels • Level 1 (Student satisfaction) • Course materials • Course activities • Instructor performance • Overall rating for course • Level 2 (Student learning) • Level 3 (Transfer of learning) • Level 4 (Cost/benefit impact)

12. Positive Student Feedback • Level 1: • Overall, students gave positive comments regarding this course. • Level 2: • The students reported a moderate level of learning. • Level 3: • Students expected professional benefits in the future from taking the course. • Level 4: • Students responded that the course cost them more money than on-campus course, but saved them time. • The majority of the students felt that the cost/benefit ratio of the course was very favorable.

13. Negative Feedback: “Distress” • Level 1: • Many students were not very satisfied with their interactions with the instructor. • Level 2: • A small number of students were not well-prepared for the technological requirements of this course, which caused them frustration. • Level 4: • Students responded the course cost them more money than an on-campus course.

14. Negative Feedback: “Distress” • Levels 1, 2, 3: • One student, in a moment of “distress,” made the following statement: "I am totally frustrated. I absolutely do not know how this class is organized and how to access the information I need. I hate Long Distance education and I never plan to do this ever again. It has made me rethink even using the Internet in my class at school. I hate this. I hate this. I hate this."

15. “Distress”: Comparison to Hara and Kling • Distress: “situations that the students...find particularly troublesome.” • Social Isolation (not found) • Overwhelming Email Communication (not found) • Lack of Instructor Feedback (found) • Technical Problems (found) • Ambiguous Instructions (found) • (Hara and Kling)

16. Recommendations for DE Instructors and Developers • The instructor should review her practices in responding to students’ email and web-based conferencing posts, to ensure that she is providing sufficient and appropriate feedback. • The instructor should specify the technological requirements in the syllabus, and arrange for technical support. • The instructor should conduct a usability tests on the course requirements and other instructions to ensure that they are clear and non-ambiguous. • The distance education program should provide an on-campus face-to-face orientation. For students that cannot be on-campus, a video tape should be provided.

17. Limitations • Small sample size (though entire class) • Education students may be atypical • Lack of follow-up evaluation • Lack of ability to measure Level 3 (Transfer of Learning)

18. Questions? Christopher Essex cessex@indiana.edu Kursat Cagiltay kursat@indiana.edu Presentation Online at: http://php.indiana.edu/~kursat/evalde-aect-denver.ppt