Measuring Teaching Quality in the Online Classroom

1. Measuring Teaching Quality in the Online Classroom Ann H. Taylor Director, Dutton e-Education Institute College of Earth and Mineral Sciences

2. How do we review teaching effectiveness at Penn State? • Per Policy HR-23 and the “Statement of Practices for the Evaluation of Teaching Effectiveness for Promotion and Tenure”: • Student reviews via the SRTE • Peer review • Procedures, including frequency, are left to academic units • “Your mileage may vary”!

3. How can we review teaching effectiveness online? • Formal reviews • Student reviews • Informal reviews

4. Are there differences in what we are looking for? Seven Principles of Good Practice Good practice in undergraduate education: • encourages contact between students and faculty, • develops reciprocity and cooperation among students, • encourages active learning, • gives prompt feedback, • emphasizes time on task, • communicates high expectations, and • respects diverse talents and ways of learning.

5. Good teaching is good teaching.But the evidence may look quite different!

7. What are we looking for? • Interactions with students (including response rates and availability) • Use of technology (including usability of instructor-created content) • Pedagogical strategies (including facilitation of discussions) • Overall management of the administrative aspects of the course

10. What should we do with the results? • Celebrate and reward good teaching • Provide professional development opportunities • Mentoring • Learning design support • Resources! (weblearning.psu.edu)

11. Student Reviews

12. What are students reviewing? • Course content • Interactions with instructors • Use of technology • Appropriateness of the technology • Pedagogical strategies

13. What are some concerns? • What can, and cannot, students evaluate? • How can you get meaningful feedback? • Are students qualified for the task? • Will they complete online evaluations? • How do I get them to fill them out?

14. What should faculty do with student feedback? • Be prepared to make a few changes • Take the small steps approach • Read results with a thick skin • Look for trends • Overlook outliers • Seek explanations, but don’t explain away (see http://www.lehigh.edu/~infdli/FD-evaluations.htm)

16. Tool 1: SEEQ Students’ Evaluation of Educational Quality • Mid-semester feedback instrument • Developed by educational psychologist Herbert Marsh • Normed to determine the kinds of questions students really could answer! • 32 standardized questions grouped into nine dimensions of teaching

17. Tool 2: SALG Student Assessment of their Learning Gains • Sample instrument that can be customized • Statements about the degree of "gain" (on a five-point scale) which students perceive they've made in specific aspects of the class • Includes open-ended questions • Delivered to students online • Sponsors provide a statistical report of the results

18. Tool 3: SRTE Student Rating of Teaching Effectiveness • 177 Question Choices • 4 mandatory items • Are you taking this course as an elective? (If uncertain, omit.) • What grade do you expect to earn in this course? • Rate the overall quality of this course. • Rate the overall quality of the instructor. • 2 open-ended mandatory items • What helped you learn in this course? • What changes would improve your learning? • Up to 15 department items and up to 5 instructor items

19. SRTE for online courses in EMS SRTE Question Bank Items: • Rate the instructor's availability for individual help and consultation.  (#42) • Rate the effectiveness of the instructor's responses to student questions.  (#48) • Rate the instructor's skill in keeping the course on schedule.  (#70) • Rate the promptness with which graded exams, reports, and other materials were returned.  (#101) Additional questions: • Rate your agreement with the following statement:  “The instructor initiated communication with students on a regular basis.” • Rate your agreement with the following statement:  “The instructor responded to my inquiries in a timely manner.” • Rate your agreement with the following statement:  “The instructor provided meaningful feedback on my class assignments.” • Rate your agreement with the following statement:  “I would recommend this course to others.”

20. Tool 4: DIY • Simple web-based form • Ask anything! • Lots of examples available • Lots of tools available • Google Template • SurveyMonkey • Survey Gold • Zoomerang

21. Another alternative:Gather informal feedback • Minute papers, muddiest points • Self-check using formal instruments • Peer Review of Online Teaching • Faculty Competencies for Online Teaching • Online Mentoring Program (pilot)

22. How do we get started? • Start with faculty • How do they define good teaching? • What criteria do they think should be measured? • How do we involve part-time faculty? • Form a steering committee of key stakeholders • Take small steps and build on successes • Measure impact throughout the process • Strive to continuously improve

23. What can administrators do? • Create an institutional environment where good teaching is highly valued & professional growth is expected • Provide adequate and accessible faculty development tools & resources • Tell faculty, staff, and students how this will benefit them • Measure impact! • Recognize and reward good teaching at all levels – celebrate! • Encourage trial and error • Act!

24. What can faculty do? • Utilize a wide array of strategies to collect feedback • Document and share evidence of good teaching with superiors • Request faculty development programming • Work with faculty senate and other faculty bodies to strengthen support • Tell your students how this will benefit them

25. So what are YOU going to do?

26. Follow-up? Ann H. Taylor atb3@psu.edu 814-863-9070