Evaluating Online Learning: Issues and Strategies

1. Evaluating Online Learning: Issues and Strategies Mark Hawkes Dakota State University Innovations, Educating New Generations March 1, 2002

2. Presentation Objective • Discuss online learning evaluation approaches in graduate programs at two universities • Identify criteria/indicators suitable for the evaluation of online learning environments

3. Distance Learning Literature Evaluation . . . . . . . No Yes (21%) (79%) Focus . . . . . . . . . Training Education (87%) (13%) Impact on Learning . . . . . . . No Yes (58%) (42%)

4. OnlineLearning Architecture Instructor Student Student Student Student Student Internet Login Interface Content Resources Management and Organizational Information Communication Modes Assessment Institutional Support Services System Resources

5. Familiar Online Learning Evaluation Targets . . . • interface design • instructional design • student satisfaction • technology access • faculty satisfaction • economic viability • departmental capacity • interdepartmental collaboration

6. Common Dimensions of Effective Online Learning: • Relevant and challenging assignments • Providing adequate and timely feedback through teacher-student interaction • Flexibility in teaching and learning • Constructing coordinated learning environments • Constructing rich environments for student to student interaction

7. DSU’s Educational Technology Program Students: • 36 Credit hour MS program • 80% Education (K12, Technical, Higher Ed.) • 20% Business/industry • 90% Online; 10% On campus • Female 68%; Male 32% • Project-based curriculum

8. DSU’s ET Environment • Pervasive technological culture • Consistency between program goals and the state/region-wide initiatives • Campus-wide faculty support • Institutional experience in Web-based instruction delivery • Multi-delivery methods • Client: teachers, teacher developers, instructors, trainers, technology coordinators, etc. • Predominantly web-based delivery

9. An Evaluation Model . . . Illuminative Operation of Components And Subcomponents Integrative Holistic perspective on The learning experience Components Infra- structure /System Course & Program Design Work Flow Interaction Impact Process Product Observing and Detecting Focused on Performance Functional Problems Outcomes

10. Infrastructure/System • Input/output devices • Network speed and connectivity • Network design/Topology • Technical support systems and maintenance

11. Course and Program Design • Nature of the Design • Situation Based • Role of State and National Standards • Sequencing/Instructional Strategies • Assessment • Motivation: Learning vs. Performance • Visualization Tools and Media • User Interface • Course Management

12. Work Flow • Use of discussion tools • Software usage • Message redundancy (audio, video, web pages, emails). • Progression • Do learners progress through their work tasks in a linear fashion? (novice-like) • Nonlinear opportunistic fashion (expert-like)

14. Interaction • Social and instructional • Must account for all of the following relationships: Instructor Learner Learners Learners Technology Technology Content Content

15. Interaction Example Learner Learner Assessment Content Learning Resources (Technology) Learner Records Instructor Performance Catalog info Query Preferences (Metcalf, Snitzer, Austin, 2001)

16. Online Course Interaction • Announcements • Email • Discussion Board (WebBoard Conferencing) • Synchronous text chat • Streaming media using Real Player • Desktop Video • File Loading • Online assessment • Audio/video clips • Audio-narrated PowerPoint's • Room-based Video

17. Impact Course performance Collaborative learning Retention/attrition (course and program) Professional relevance and utility Learner productivity

18. Evaluation Attributes • Multi-sourced data (students, server log files, etc) • Internal and external • Performance based • Comparison and criterion based

22. The breadth of this course was: 1 2 3 4 5 6 7 Not nearly enough The right amount Way too much Compared to a traditional course 1 2 3 4 5 6 7 A much narrower range About the same range A much wider range of of material was covered of material was covered material was covered Online: 4.61 Compared to traditional 4.65 n=32

23. The depth of this course was: 1 2 3 4 5 6 7 Not nearly enough The right amount Way too much Compared to a traditional course 1 2 3 4 5 6 7 Material was covered in Material was covered in Material was covered in much less depth about the same depth much more depth Online: 4.48 Compared to traditional 4.42 n=32

24. The extent of critical thinking required: 1 2 3 4 5 6 7 Not nearly enough The right amount Way too much Compared to a traditional course 1 2 3 4 5 6 7 Much less About the same Much more Online: 4.61 Compared to traditional 4.94 n=32

25. The amount of effort put into the course: 1 2 3 4 5 6 7 Much less About the same Much more Compared to a traditional course 1 2 3 4 5 6 7 Much less About the same Much more Online: 5.65 Compared to traditional: 5.26 n=32

26. Evaluation Processes • Mostly formative • Mixed data collection and analysis methods (document analysis, student artifact analysis, survey, interview, text analysis, etc.) • Course and program evaluation

27. Typical Problems with Online Courses • Facilitating and encouraging collaboration • Time management • Student proficiency with course tools • Ambiguous directions • Timeliness of feedback

28. Factors Beyond ID Control • Student sophistication with technology tools • System capacity • Learner availability/accessibility • Enthusiastic, responsive instructor • Good learner support • Motivated learners

29. How to Design and Effective Online Course? • Follow basic ID principals • Build a climate of disclosure and full participation • Institute informal student evaluation and check-in mechanisms • Active and intensive instructor participation • Build in as much interactivity as possible • Create visually interesting screens/pages • Ensure instructions are very clear • Multi-mode interaction is critical • Redundancy