Coulter Faculty Center eLearning Faculty Fellows at Western Carolina University

1. An Online Course Assessment Tool (OCAT)for Faculty Peer-assessment or Self-assessmentSpeaker: Claire DeCristofaro, MDAssoc Prof, WCU Nursing, MS(N) ProgrameLearning Faculty Fellow, Coulter Faculty CenterWestern Carolina University, Cullowhee, NC Sloan-C ConferenceNovember 2007

2. Coulter Faculty Center eLearning Faculty Fellowsat Western Carolina University • eLearning Faculty Fellows • John LeBaron (Distinguished Professor, Education) • Claire DeCristofaro (Assoc Prof, Nursing) • Dixie McGinty (Assoc Prof, Education) • Mary Teslow (Asst Prof, Health Sciences)

3. Why perform peer assessment? • Provide constructive feedback regarding teaching effectiveness • Stimulate instructional improvement • Promote faculty development • Provide opportunities for peer support

4. Online Course Assessment Tools • Open source resources & rubrics • Universities (course reviews) • Quality Matters, Sloan Consortium • Different formats, priorities, methods • Our tool: • Blends recurring themes • Includes narrative summaries • Accommodates varied disciplines • Accommodates varied student cohorts • Includes instructor feedback

5. OCAT Reliability Testing • IRB-approved research project to determine inter-rater reliability • Tool used to assess existing courses drawn from multiple disciplines • Courses used for reliability testing selected because instructors are experienced online teachers or instructors have an interest in online teaching • Reliability project included peer assessors drawn from faculty across disciplines and from instructional design staff at the faculty center

6. Peer Assessors • Will have been trained in the use of the tool • Will honor the confidential nature of this activity • Will maintain confidentiality of results

7. Overview of the Tool • Interactive PDF document • Process Data • First five sections cover specifed domains • Course & instructor assessment • Each domain includes assessment of: • Elements of course design • Elements of teaching • Last two sections • Peer Assessor summary narratives • Instructor response to assessment process & assessment results

8. Process Data • Instructor (rank & department) • Course (code, section, term) • Peer Assessor (rank, department) • Format of CMS • Dates – request, assessment, completion • Number of students in attendance

9. Performance Levels for Rated Elements • “Scoring” for listed elements: • Exemplary: exceeds the element • Standard: meets the element • Developing: falls beneath the element statement • Not Applicable: not relevant to the course • Includes Narrative Commentary boxes for each section • Getting “Dinged” ??

10. Course & Instructor Domains • Course Overview & Organization • Learner Objectives & Competencies • Resources & Materials • Learner Interaction • Learner Assessment

11. Final Narrative Summaries 6. Peer Assessor narrative commentary • Additional comments on strengths of course • Additional comments regarding recommendations for improvement • Recommendations for improvement 7. Instructor response narrative commentary • Suggestions for altering peer assessment process • Suggestions for revising the peer assessment process

12. Confidential Peer Assessment Process (1) • Faculty requests peer assessment of online course through the CFC • CFC Director arranges peer assessment (selects peer assessor) • Trained peer assessors meet with instructor before beginning assessment

13. Confidential Peer Assessment Process (2) • Peer assessor completes sections 1- 6 (peer assessor sections) • Peer assessor meets again with instructor and section 7 is completed (instructor feedback section) • Final, completed document is sent to instructor and also kept in confidential file at the faculty center

14. Use of OCAT for faculty self-assessment • The peer assessment instrument itself will also be available online for faculty • The tool can be used for self-assessment as a faculty development tool • Additionally, the tool can be used as a “guide” for course development, especially for faculty new to the online teaching environment

15. Faculty Feedback on OCAT • OCAT has been presented to several university workshops • Faculty have had the opportunity to: • use the tool to evaluate their own courses • use the tool in evaluating courses volunteered by experienced online teachers • Faculty response overall positive – tool provides: • a “guide” to effective online teaching practices • a template for course development • a structured guide for ongoing improvement of existing courses for serial re-evaluations over time

16. Domain 1Course Overview and Organization

17. A. Elements of Course Design • Navigational instructions make the organization of the course easy to understand and are transparent at the beginning of course • Course introduction that includes guidance on the structure of the course is provided • Course design demonstrates user-friendly presentation style • Minimum student technology competencies/skills for the course are stated, if applicable • Course-specific technology requirements are stated, if applicable • Description and method of accessing the institution’s academic support system is defined • The institution’s support services and resources (e.g. library) are described and linked appropriately • Course instructions articulate or link to tutorials and resources that answer basic questions related to research, writing, technology, etc. • Course design item(s) emerging from peer discussion not included in the list above (type in box below):

18. B. Elements of Teaching: The instructor: • provides an introduction and provides credentials related to this course • provides the opportunity for student introduction • provides the opportunity to practice and master the technologies needed for the course • assures that learners are ready to undertake assigned tasks • states the broader scholarly/practical context for learning objectives/goals • Instructional item(s) emerging from peer discussion not included in the list above (type in box below):

19. Domain 2Learner Objectives and Competencies

20. A. Elements of Course Design Learning Objectives/Goals: • are stated for each unit/module • are clear and easy to understnd • describe outcomes that are assessable • address content mastery • address critical thinking skills • address core learning skills • Course design item(s) emerging from peer discussion not included in the list above (write in box below):

21. B. Elements of Teaching The instructor: • provides information to students on how to meet the learning objectives/goals • explains how to meet the learning objectives/goals in a manner that is clear and easy to understand • fosters integration with prior learning, if applicable • Instructional item(s) emerging from peer discussion not included in the list above (type in box below):

22. Domain 3Resources & Materials

23. A. Elements of Course Design Resources and Materials: • address learning objectives/goals (e.g. course reference materials, glossaries, library resources, relevant web resources, discipline-specific resources • are appropriatly authenticated and cited • are current and/or timely • are prepared by qualified sources • are presented in a format appropriate to the online environment • are easily accessible to the student • state the purpose related to the course (e.g. core material, supplemental, historical, required/optional) • have active links to electronic resources • include the use of electronic library scholarly resources, if applicable • address diverse learning styles (e.g. Multimedia, graphics, audio, video, text chats, voice chats • Course design item(s) emerging from peer discussion not included in the list above (type in box below):

24. B. Elements of Teaching The instructor: • Demonstrates knowledge of the discipline • Incorporates strategies that promote the progressive learner construction of knowledge (e.g. use of units/modules that are thematically connected and have a logical progression) • Encourages students to independently augment the course resource base through independent or collective research, if applicable • Instructional item(s) emerging from peer discussion not included in the list above (type in box below):

25. Domain 4Learner Interaction

26. A. Elements of Course Design • The course design prompts the instructor to be present, active and engaged with the students (e.g. provides tools necessary) • Netiquette instructions/recommendations provided regarding emails and discussion postings • Guidelines for collaboration are defined, if applicable • Guidelines for communication/accountability are defined • Course design item(s) emerging from peer discussion not included in the list above (type in box below):

27. B. Elements of Teaching The instructor: • Is aware of student progress (e.g. assigned tasks) and any barriers to progression • Fosters interaction among constituencies inside and outside the course as appropriate (e.g. student-student, student-instructor, and with external persons or agencies • Embeds learning activities (student-content interaction) that promote the achievement of stated objectives and learning outcomes • Selects communication strategies to communicate appropriately with the class as a whole, student groups, and individuals • Respects student privacy as necessary • Promotes independent and shared student research, if applicable • Provides content-based interaction as needed • Provides timely responses to student queries • Offers announcements tailored to the progress of the class • Models a communication style that demonstrates a positive tone • Instructional item(s) emerging from peer discussion not included in the list above (type in box below):

28. Domain 5Learner Assessment

29. A. Elements of Course Design • Assessments are frequent enough to provide formative feedback • Rubrics are provided to define assessment criteria, if applicable • Methods of submitted assignments are appropriate to the online learning environment • Course design item(s) emerging from peer discussion not included in the list above (type in box below):

30. B. Elements of Teaching The instructor: • Clearly describes assignments (student discussion, participation, and projects) • Provides formative assessment/progress feedback • Provides summative assessment feedback • Provides timely assessment feedback • Instructional item(s) emerging from peer discussion not included in the list above (type in box below):

31. Section 6:Peer Assessor Commentary – Final Narrative Summaries 1. Additional comments regarding the strengths of this course and teaching (type in box below):2. Additional comments regarding recommendations for improvement (type in box below):

32. Section 7:Instructor Response – Final Narrative Summaries • Instructor response to the results of this assessment (type in box below): • Suggestions for revising the peer assessment process (type in box below):

33. The Dynamic OCAT • Current length of document in order to allow extensive narrative summaries in each section • Modifications planned based on implementation and feedback • modify assessment process & data collection • modify OCAT itself

34. Thank youComments? Questions?You are also welcome to email:decristofaro@email.wcu.edu