Using the Quality Matters Rubric to Improve Online Courses ed bowen, Executive Dean Distance Learning, Dallas TeleColleg

1. Purpose of Training (QM Reviewers & supporters) Elm ForkPurpose of Training (QM Reviewers & supporters) Elm Fork

2. Training session objectives to review with the participants. NOTE: The day�s evaluation is based on how well the participants feel that these objectives were met.Training session objectives to review with the participants. NOTE: The day�s evaluation is based on how well the participants feel that these objectives were met.

3. Marker: The following slides deal with the �key� QM concepts.Marker: The following slides deal with the �key� QM concepts.

4. QM Continuous Improvement Process: This visual highlights the QM review process. Note that the circle elements transition in automatically. Key points: Designed for continuous improvement; the goal is that ALL courses will eventually meet QM expectations. Method: Begin with the COURSE and then continue through PEER COURSE REVIEW, FEEDBACK, COURSE REVISION and Course Meets Quality Expectations. Speak briefly about each item and how it contributes to the process. Reassure participants that they will more about each major item in detail throughout your presentation.QM Continuous Improvement Process: This visual highlights the QM review process. Note that the circle elements transition in automatically. Key points: Designed for continuous improvement; the goal is that ALL courses will eventually meet QM expectations. Method: Begin with the COURSE and then continue through PEER COURSE REVIEW, FEEDBACK, COURSE REVISION and Course Meets Quality Expectations. Speak briefly about each item and how it contributes to the process. Reassure participants that they will more about each major item in detail throughout your presentation.

5. Underlying Principles of QM The QM toolset and process are: based in national standards of best practice, the research literature and instructional design principles designed to promote student learning integral to a continuous quality improvement process part of a faculty-driven, peer review process Course does not have to be �perfect� but better than just �good enough.� (Standards met at about 85% level or better.) ****Direct participants to a copy of the Underlying Principles handout, located in the front pocket of the Binder**** . In early iterations of the Peer Reviewer process, many participants focused on HOW to do the review, but missed the WHY. This slide emphasizes the key concepts behind QM and how they contribute to a quality course. Go over each one�you may want to direct participants to the Research Matrix, located on the web and as a print copy behind Tab 5 of the Binder. The Research Matrix provides citations for the national standards and research literature used to develop the rubric.****Direct participants to a copy of the Underlying Principles handout, located in the front pocket of the Binder**** . In early iterations of the Peer Reviewer process, many participants focused on HOW to do the review, but missed the WHY. This slide emphasizes the key concepts behind QM and how they contribute to a quality course. Go over each one�you may want to direct participants to the Research Matrix, located on the web and as a print copy behind Tab 5 of the Binder. The Research Matrix provides citations for the national standards and research literature used to develop the rubric.

6. Underlying Principles of QM Continued Process designed to ensure all reviewed courses will eventually meet expectations Collegial review process, not an evaluation process Review team must include an external peer reviewer Course faculty or instructor considered part of the review team Emphasize the first bullet: If the QM process is followed, ALL courses will meet expectations. This is an important concept that new Peer Reviewers often miss. Emphasize the first bullet: If the QM process is followed, ALL courses will meet expectations. This is an important concept that new Peer Reviewers often miss.

7. What�s In It For Institutions � External validation Strengthen accreditation package Raise QA as a priority activity Access to a sustainable, replicable, scalable QA process Inform online course training & practices Provide professional development activities

8. What�s In It For Faculty � Improve online courses External quality assurance Expand professional community Review other courses and gain new ideas for own course Participation useful for professional development plan and portfolio Receive $150 for each completed peer course review

10. Training session objectives to review with the participants. NOTE: The day�s evaluation is based on how well the participants feel that these objectives were met.Training session objectives to review with the participants. NOTE: The day�s evaluation is based on how well the participants feel that these objectives were met.

11. Quality MattersPeer Course Review Process Briefly go over the steps in the Peer Course Review Process.Briefly go over the steps in the Peer Course Review Process.

12. Quality Matters Peer Course ReviewerCertification Process Emphasize that the participants in today�s group is now TRAINED. Each participant will become certified only when he/she completes an actual QM Peer Review. If participants ask how to become Peer Reviewers, direct them to the QM web site.Emphasize that the participants in today�s group is now TRAINED. Each participant will become certified only when he/she completes an actual QM Peer Review. If participants ask how to become Peer Reviewers, direct them to the QM web site.

13. The Rubric Eight general standards: Course Overview and Introduction Learning Objectives Assessment and Measurement Resources and Materials Learner Interaction Course Technology Learner Support Accessibility Introduce the 8 main sections of the rubric (these are the General Review Standards and are noted in Roman numerals on the rubric). Point out that QM tries to take a holistic view of the course and that�s why it�s so important that the Learning Objectives ALIGN with the Assessments, Resources and Materials, Interaction, and Technology. Take time to point out how the rubric is organized: Eight General Review Standards Forty Specific Review Standards In Table Format: 1st column is Specific Review Standards, 2nd column is point value, and 3rd column is annotation.Introduce the 8 main sections of the rubric (these are the General Review Standards and are noted in Roman numerals on the rubric). Point out that QM tries to take a holistic view of the course and that�s why it�s so important that the Learning Objectives ALIGN with the Assessments, Resources and Materials, Interaction, and Technology. Take time to point out how the rubric is organized: Eight General Review Standards Forty Specific Review Standards In Table Format: 1st column is Specific Review Standards, 2nd column is point value, and 3rd column is annotation.

14. Key Standards That Must Align Objectives Standard II.1: Measurable outcomes Standard II.2: Content mastery, critical thinking, & learning skills addressed. Assessment and Measurement Standard III.1: Measures objectives; consistent with learning activities Learner Interactions and Activities Standard V.1: Help students achieve the objectives Course Materials Standard IV.1: Deep and comprehensive enough for students to achieve the objective Course Technology Standard VI.1: Tools and media support the objectives Again, the General Standards and the 3-point Specific Review Standards that must align.Again, the General Standards and the 3-point Specific Review Standards that must align.

15. General Standard I:Course Overview and Introduction 1.1: Navigational instructions make the organization of the course easy to understand. 1.2: A statement introduces the student to the course and to the structure of the student learning, and, in the case of a hybrid course, clarifies the relationship between the face-to-face and online components. 1.1: Navigational instructions make the organization of the course easy to understand. Another way of writing this standard might be: Do students know what to do first? It�s similar to the type of information provided in the Schedule of Classes (dates, days, time, classroom, instructor, etc.) Your college would never just give students the name of the class without the details and expect them to arrive at the right classroom at the right time on the right day. Remember that you are taking the student�s point of view (not the instructor�s) and you should not assume information that is not clearly stated. If YOU can�t find the information or are confused about what to do first, the students taking this course will also be confused. 1.2: A statement introduces the student to the course and to the structure of the student learning, and, in the case of a hybrid course, clarifies the relationship between the face-to-face and online components. Students new to this course should not have to guess about how the course is organized, what learning activities are required and which are optional, how to communicate with the instructor and fellow students, and other general course information. This is the type of information usually found in the syllabus and that the instructor goes over on the first meeting of a face-to-face class. The online instructor should provide this information in a clearly stated and easy to find way, taking care to focus on those aspects of the course that will be of primary interest to students (due dates, late policy, email policy, etc).1.1: Navigational instructions make the organization of the course easy to understand. Another way of writing this standard might be: Do students know what to do first? It�s similar to the type of information provided in the Schedule of Classes (dates, days, time, classroom, instructor, etc.) Your college would never just give students the name of the class without the details and expect them to arrive at the right classroom at the right time on the right day. Remember that you are taking the student�s point of view (not the instructor�s) and you should not assume information that is not clearly stated. If YOU can�t find the information or are confused about what to do first, the students taking this course will also be confused. 1.2: A statement introduces the student to the course and to the structure of the student learning, and, in the case of a hybrid course, clarifies the relationship between the face-to-face and online components. Students new to this course should not have to guess about how the course is organized, what learning activities are required and which are optional, how to communicate with the instructor and fellow students, and other general course information. This is the type of information usually found in the syllabus and that the instructor goes over on the first meeting of a face-to-face class. The online instructor should provide this information in a clearly stated and easy to find way, taking care to focus on those aspects of the course that will be of primary interest to students (due dates, late policy, email policy, etc).

16. Rubric Scoring Point out the breakdown of the 40 Specific Review Standards by points: 14 Standards are ESSENTIAL. Without all standards being met at the 85% level, this course cannot be a quality online course. 12 Standards are Very Important to an online course 14 Standards are Important. This results in a total of 40 Specific Review Standards and a total of 80 points. Emphasize again that each reviewer individually decides Yes or No on a particular standard and enters this decision on the web form. The web form then automatically totals the points (majority rules) and determines whether/not the course meets the standard.. Point out the breakdown of the 40 Specific Review Standards by points: 14 Standards are ESSENTIAL. Without all standards being met at the 85% level, this course cannot be a quality online course. 12 Standards are Very Important to an online course 14 Standards are Important. This results in a total of 40 Specific Review Standards and a total of 80 points. Emphasize again that each reviewer individually decides Yes or No on a particular standard and enters this decision on the web form. The web form then automatically totals the points (majority rules) and determines whether/not the course meets the standard..

17. The Peer Review Team 3 Faculty Peer Reviewers All must be experienced online instructors All must attend QM training One MUST be external to the course developer�s institution One must be a subject matter expert (SME) The SME could also be the external reviewer. AND Faculty Course Developer access to rubric prior to review involved in pre-review discussions consulted during review Be sure to point out that there a 3 faculty peer reviewers who actually complete the web-based review form. One of these reviewers must be a Subject Matter Expert and at least one must be external to the course�s home institution. Emphasize that the Faculty Course Developer (the Instructor) is a part of the review team and should be consulted by the team. Have any of you developed an oline course? Be sure to point out that there a 3 faculty peer reviewers who actually complete the web-based review form. One of these reviewers must be a Subject Matter Expert and at least one must be external to the course�s home institution. Emphasize that the Faculty Course Developer (the Instructor) is a part of the review team and should be consulted by the team. Have any of you developed an oline course?

18. Faculty Developer Part of the review team. Provides access to the course. Completes Instructor Worksheet Part of the initial team discussion Receives compiled report Returns Faculty Response Form Emphasize that the faculty course developer is a part of the review team and should be consulted when questions or concerns arise. It is perfectly fine � and encouraged � if the faculty member wishes to make revisions before the review is complete (QM only wants a better course; this is not a faculty review). For example, if the Team doesn�t find a statement directing students to campus ADA services (Standard VIII, 3-point essential), ask the faculty developer if it�s there and, if it�s not, to put it in now rather than waiting. This immediately improves the course AND may make it possible for the course to meet expectations on the initial review. If you have not pointed out the examples of the compiled reports, Sample Final Review Reports (behind Tab 4 in the Binder), do so now so that the peer reviewers know what these reports look like. The Faculty Response Form (behind Tab 4 in the Binder) gives faculty choices depending on whether their course met or did not meet expectations. Emphasize that the faculty course developer is a part of the review team and should be consulted when questions or concerns arise. It is perfectly fine � and encouraged � if the faculty member wishes to make revisions before the review is complete (QM only wants a better course; this is not a faculty review). For example, if the Team doesn�t find a statement directing students to campus ADA services (Standard VIII, 3-point essential), ask the faculty developer if it�s there and, if it�s not, to put it in now rather than waiting. This immediately improves the course AND may make it possible for the course to meet expectations on the initial review. If you have not pointed out the examples of the compiled reports, Sample Final Review Reports (behind Tab 4 in the Binder), do so now so that the peer reviewers know what these reports look like. The Faculty Response Form (behind Tab 4 in the Binder) gives faculty choices depending on whether their course met or did not meet expectations.

19. Peer Reviewers Establish Team Calendar Review the course individually Complete the online web review form Discuss review with Team as needed Complete an Exit Interview Receive� Recognition as Certified Peer Reviewer. Compensation Again, the expanded version of Peer Review Roles and Responsibilities is behind Tab 4 in the Binder. Again, the expanded version of Peer Review Roles and Responsibilities is behind Tab 4 in the Binder.

20. Team Chair Reviewer Roles & Responsibilities plus�. Organizes Team calendar Confirm Instructor Worksheet is used Creates draft report from compiled reviews Convenes Team discussions Reviews, edits and submits Team Report Receives� Recognition as Certified Peer Reviewer and Chair. Compensation The Team Chair has additional responsibilities and is usually receives slightly more compensation. The Team Chair organizes the calendar, convenes Team Discussions, and edits the Team Report. Note here that the actual �compiling� of the Team Report is done automatically by the web-form. The Team Chair is mostly reviewing for errors and making sure that there are explanations for all �NO� scores.The Team Chair has additional responsibilities and is usually receives slightly more compensation. The Team Chair organizes the calendar, convenes Team Discussions, and edits the Team Report. Note here that the actual �compiling� of the Team Report is done automatically by the web-form. The Team Chair is mostly reviewing for errors and making sure that there are explanations for all �NO� scores.

21. Overall Course Review Results Upon initial review: 51% meet expectations 19% do not meet expectations - missing at least one essential 3-point element(s) 30% do not meet expectations - missing at least one essential 3 point element(s) and a minimum of 68 points

22. Training session objectives to review with the participants. NOTE: The day�s evaluation is based on how well the participants feel that these objectives were met.Training session objectives to review with the participants. NOTE: The day�s evaluation is based on how well the participants feel that these objectives were met.

25. Additional Optional Services and Fees: Independent (Subscriber-Run) Peer Reviews Subscribers may manage their own MOL-recognized Peer Reviews after training. Institutions are responsible for selecting and managing peer reviewers per Quality Matters program guidelines and for compensating peer reviewers. Subscribers are given access to the Quality Matters Rubric for the review team and recognition in the Quality Matters� Registry of courses that meet standards. The annual subscriber fee includes registration of up to 30 independent reviewed courses for Large Institutions, 20 for Mid-Sized Institutions, and 10 for Small Institutions. (Subscribers pay a $50 per-course registration fee for any additional courses.) To be eligible for recognition in the Quality Matters� Registry, an on-line course must have successfully completed a subscriber-run review that has met all Quality Matters guidelines (which may or may not involve additional costs to the Institution). Guidelines include: Review managed by a QM-trained Institutional Representative (one seat in training included in annual subscriber fee) Review completed by a review team of three Qualified Quality Matters-Trained Reviewers. Team make-up must include a reviewer from outside the institution; a reviewer trained as a Quality Matters Master Reviewer; and a reviewer assigned as team chair. (A single reviewer may fill one or all of these roles.) Reviewers used the Quality Matters materials and followed the Quality Matters process (provided electronically to subscribers as part of annual subscriber fee) Upon successful completion of review and submission of documents to MOL, course institution has paid $50 per course for each registration over those included in annual subscriber fee MOL provides access to list of trained Peer Reviewers to subscribers. Subscribers must negotiate the peer reviewer stipends paid if any. (MOL pays $150 per reviewer, $250 to team chair)

26. MOL-Managed Peer Reviews MOL will manage the course review process for subscribing and non-subscribing institutions on a course-by-course basis. This service includes: Peer reviewers assigned, managed, and paid by MOL; compiled final report; and recognition of a course in the Quality Matters Registry upon its meeting Quality Matters program review standards. Quality Matters Program Subscribers $750/course Non-Subscribers $1,000/course

28. Multiple Uses of QM Guidelines for initial online course development Quality assurance of existing courses Ongoing faculty professional development Institutional reaccredidation packages Formation of distance learning policies & steering committees