The Seventh Sloan-C International Conference on Online Learning

1. The Seventh Sloan-C International Conference on Online Learning November 16-18, 2001-- Orlando, Florida A Model for Peer Review Process Of Web-based Courses

2. Order of the Presentation Introduction Questions ? Background The Model The Process Categories& Criteria Conclusion

3. Background WIT • Developing Criteria • Piloting Courses • Dissemination • 13 USM Institutions • Web-based Pedagogy • Faculty Interaction • Online Course Development

4. Quality Concerns • AAUP’s position:  • D/E courses shall comply with all requirements for F2F • Faculty involvement at course development, approval, etc. • Faculty approval of rules governing D/E & its technologies • CHEA’s position: • Assessment process of D/E must be driven by faculty • Proposes 8 Conditions to assure quality • Faculty’s position: • Online courses have a disregard for depth of inquiry • Weak access to learning resources and support services PRS will address these concerns

5. CHEA's Conditions • Establish performance measures with evidence for reliability and validity for distance learning • Require providers to substantiate evidence of contact between faculty and students • Require evidence of effective instructional techniques • Promote systematic efforts for selecting and training faculty • Assure the availability of learning resources • Promote ongoing monitoring and enhancement of the technology infrastructure of institutions • Focus attention on the development of courseware and the availability of information • Examine alternatives to the traditional accreditation process.

6. Culminating in The Model • Peer Review System (PRS) • Cross-institutional • Faculty driven • Collaborative • Literature-based • Empirically tested • Scalable Development of Quality Assurance Criteria

7. Categories Design Assessment Delivery

8. Criteria Design • Pedagogy • Course Content/objectives (Khan, 1997) • Engagement (keep the learners interest) • Learner centered Assignments (clarity about the scope & form) • Instructor’s feedback, mentoring, motivation, and facilitation (Overmyer, 1998)

9. Criteria Design • Use of media in support of learning and learning styles • Use of audio and video clips • Use of simulation • Conferencing (synchronous, asynchronous) • Interactivity • Learner/learners/moderator • Learner/course material-assignments • Learners/resources (library; advising; academic support services) • Sense of belonging to a learning community

10. Criteria Design • Navigation • Avoid long scrolling pages • Avoid constantly running animation on pages • Avoid complex URL • Create a link to home page from every page • Create links among similar and related concepts (ideas) • Avoid non-standard link colors • Weed out outdated information • Include a search feature • Be concerned about the bandwidth

11. Criteria Design • Use of WWW resources (to meet course objectives) • Relevance • Suitability of the material ((Khan, 1997) • Sustainability • Web dependent vs. software dependent contents

12. Criteria Delivery • Experience of the moderator with teaching online • Support system • Academic • Administrative • Technical • Contingency strategies (other communication modes)

13. Criteria Assessment • Students • Instrument (online/monitored quiz or exams, posted or emailed assignments) • Grading techniques (learner or moderator scored, software scored; other arrangements) • Authenticity of student's work

14. The PRS Approach The Process • Current State of Web-based Course Development • Initiatives of individual faculty • Campus sponsored workshops • Outsourcing • Cross-institutional approach

15. Criteria Developed from 86 suggested Variables by WIT Participants WIT Criteria • Syllabus • Learner-centered approach, design • Instructional design/learning strategies • Interactivity • Use of Media • Use of WWW • Internal consistency and clarity • Assessment • User interface, graphic design, formatting • Navigation • Access, technical robustness and student support

16. The Process • Evaluation Paradigms • Objectivist • measures outcomes against pre-set standards • Relativist • measures the outcomes of Web-based education against traditional education • Developmental • address the weaknesses of such courses and provide solutions for those shortcomings • The Four Phases • Phase 1 • development of “rough” variables • Phase 2 • design of the a survey instrument • Phase 3 • analyzing and comparing the data • Phase 4 • development of the final set of criteria

17. The Process Factor Analysis • Comparison between survey results by the “course monitors” and students • Calculation of the correlation matrix among the variables for each group • Factor analyze each correlation matrix to determine how high each variable loads on the criterion (factor) under which the variables are clustered. • This will allow condensing the original variables into a smaller set of variables under each criteria. • A comparison of the variable loading for the two groups would reveal whether or not both groups have equally signified the importance of the variables in the assessment of the online course. • This approach will further allow drawing conclusions on the significance of each criterion (factor) in explaining the quality and effectiveness of the courses.

18. Concluding Remarks • The PRS would improve upon current practice in several ways: • Faculty would have clear, concise, empirically validated criteria for quality Web-based courses • The collaborative nature of PRS allows faculty to interact with their peers and draw upon their collective experience to further enhance the quality of their online courses • Courses reviewed under the PRS will have the imprimatur of an independent body attesting to their quality • Prospective students shopping for courses can select from among those that have been reviewed under the PRS

19. Concluding Remarks • Faculty who has developed these courses can receive recognition for their superior efforts. When the faculty is considered for merit raises, promotion, or tenure, this seal of approval will attest to their courses as refereed work • Quality courses will justify expenditures on development of distance education programs • Quality courses will help facilitate the transference of academic credits between institutions • Likewise, accrediting bodies such as the AACSB, NCATE, and others can benefit from a set of widely accepted criteria for quality