Evaluating Online Courses

1. Evaluating Online Courses

2. What Is Evaluation? Per Michael Scriven (1991): • “Process of determining the merit, worth, or value of something” • “(Process that) leads to evaluative conclusions” • good/bad • works/ doesn’t work • high/low quality

3. Course Evaluation • Is the course working as intended? • Does the course meet its objectives? • Is the online learning environment as effective as it can be? • Is the course manageable? • Are learners satisfied?

4. Why Evaluate? • To improve instructional quality • To motivate learners • To motivate instructors *Scriven, M. (1991). Evaluation Thesaurus, 4th Ed. Sage Publications: London

5. Conducted by Administration vs. Conducted by Instructor • Administration-Led • Focuses on learner satisfaction & broader issues of program • Access, Course Management System • University service – registration, instructor • Minimal instructor feedback • Instructor-Led • Focuses on course components/design • Student feedback & performance • Instructor reflection

6. Needs. Anal. Revise Per Obj. Inst. Strat. Inst. Mtls. Form Eval. Assmt. Instr. Goals Lnr. & Context Summ Eval. Dick and Carey Design Model

7. Evaluate Perform. Assess Entry Behavior Analyze Feedback Gerlach and Ely Design Model Determine Strategies Specify Content Org. Groups Allocate Time Specify Object. Allocate Space Select Resources

8. In Reality, Evaluation is…. Done continuously once the course begins.

9. Recommendation Formativecourse evaluation conducted by the Instructor.

10. It needs to be… Informal and Reflective, as opposed toReactive.

11. Interim vs. Terminal • Interim • Evaluating at various points in the course • Not TOO many points  • 2-3 times per semester, mid-term at the minimum • May affect future weeks of course • Terminal • Conducted at the end of the course • Affects next course offering • May get more input on interim evaluations than terminal evaluation

12. Evaluation Framework

13. AEIOU – Course Evaluation(Based on AEIOU Program Evaluation Method) • A – Access – Was the course easily and consistently accessible? • E – Effectiveness - How well was the course designed/organized? Objectives & instructions clear? Assessment matched objectives? Level of student performance? • I – Interaction – Was there a good balance of interaction? Level of learner participation in discussions? Level of peer support, instructor support? • O – Obstacles – What hindered the potential effectiveness of the course? • U – Unanticipated Consequences – What happened that was not expected --- both from learner and instructor perspective?

14. Evaluation Considerations • Environment • Course structure • Access reliability • Instructional Strategy • Objectives • Activities • Interaction/Communication • Assessment

15. Learner Performance Motivation Satisfaction Instructor Performance Motivation Satisfaction Evaluation Considerations

16. Better to Design the Quality In Than to evaluate the problems out

17. Communication Tools for Evaluation • Email • Survey attached • Quick/one question surveys • Online surveys • Quick feedback on a new technique – “What did you think? Check the box..” • Discussion board (evaluation summaries) • Fax • For ‘handwriters’

18. Encouraging Completion – Interim • Make it as short/easy as possible • Students need to trust interim won’t affect their grade – stay away from instructor components (i.e. feedback) • Evals could be submitted to peer/TA who summarizes and forwards to instructor or posts on course site • Could be incremental questioning • Every student to respond at least once

19. Encouraging Completion - Final • One peer ‘holds’ evals until after final grades are submitted • Could be team leaders • Make it the final assignment and student would have to turn it in (to peer/TA) before final grade was submitted

20. Summary • Clear, easy, non-threatening – to maximize feedback • Focus on key components of your course and key participants in the course • Include YOUR thoughts as well as your students