Implications of Web-based Learning for Student Evaluation of University Teaching

1. Implications of Web-based Learning for Student Evaluation of University Teaching Chuck Dziuban Steve Sorg Research Initiative for Teaching Center for Distributed Effectiveness Learning Ida Cook Morgan Wang Department of Sociology Department of Statistics Patsy Moskal Research Initiative for Teaching Effectiveness University of Central Florida

2. Students Faculty Quality assurance Success rates Attitudes Online surveys Modified instructional theories Demographic inertia Withdrawal rates Large online classes Strategies for success Reactive behavior patterns Online programs Accreditation Distributed Learning Impact Evaluation components

3. Rationale for the Study • Teaching evaluation data for a 3-year period were available to allow a comparison of two different sets of items (UCF/BOR).  • Responding to faculty interest, the UCF Faculty Senate requested that an evaluation of the Student Evaluation of Instruction measures be performed. (FS 1995-96-11)

4. The Instrument: UCF Items • Feedback on your performance in this course • The instructor’s interest in your learning • Use of class time • The instructor’s overall organization of the course • Continuity from one class meeting to the next • The pace of the course • The instructor’s assessment of your progress • The text and supplemental learning materials used

5. The Instrument:Board of Regent Items • Description of course objectives and assignments • Communication of ideas and information • Expression of expectations for performance • Availability to assist students In or outside of class • Respect and concern for students • Stimulation and interest in the course • Facilitation of learning • Overall assessment of instructor

6. Approximately 450,000 student responses Five Colleges: Arts and Sciences Business Administration Education Engineering Health and Public Affairs The Study Layout • Three Levels: • Lower Undergraduate • Upper Undergraduate • Graduate • Three Years: • 1996-97 • 1997-98 • 1998-99

7. Findings Summary • Correlations among the 16 items are high -- median is approximately .70 • Correlation between UCF and BOR forms = .92 • Reliabilities of UCF & BOR forms are high • There is only one factor • Overall rating of the instructor is most strongly related to other items. • The items have a characteristic distribution

8. Squared multiple correlations from each item with the remaining ones Item W F2F 1 .75 .69 2 .77 .74 3 .74 .69 4 .72 .76 5 .75 .71 6 .64 .66 7 .77 .75 8 .50 .49 Item W F2F 9 .72 .71 10 .80 .78 11 .78 .76 12 .70 .63 13 .78 .72 14 .79 .75 15 .82 .76 16 .88 .85

9. Variance components (Generalizability Theory) for the UCF items W F2F Variance Variance Component% Component % Students .726 67.85 .728 65.23 Items .005 00.47 .002 00.19 Error .339 31.68 .386 34.58

10. Variance components (Generalizability Theory) for the BOR items W F2F Variance Variance Component% Component % Students .799 75.66 .855 71.96 Items .005 00.48 .008 .0069 Error .252 23.86 .325 27.35

11. Overall Rating of the Instructor(N=444,017) 43% 29% 19% 7% 2% Excellent Very Good Good Fair Poor

12. Student Ratings by Modality Very Modality Excellent Good Good Fair Poor F2F 42.00 29.50 19.00 7.20 2.40 (N=628,623) E 44.00 29.10 17.40 6.90 2.60 (N=6,632) M 40.60 28.60 20.60 7.70 2.40 (N=11,450) W 55.40 25.20 12.10 4.90 2.50 (N=5,435) ITV 20.90 26.20 30.50 16.50 5.90 (N=3,218)

13. A comparison of W and F2F percentage of “excellent” ratings on UCF items W (Fully online courses) F2F (Face-to-face courses) 52% 51% 50% 47% 41% 40% 37% 27% Item 2 Item 3 Item 5 Item 8

14. A Comparison of W and F2F percentage of “excellent” ratings on BOR items W (Fully online courses) F2F (Face-to-face courses) 59% 55% 52% 50% 47% 42% 38% 34% Item 12 Item 13 Item 15 Item 16

15. Overall Rating of the Instructor Binary Decision Tree SAS Enterprise Miner Predictors • Level • Year • College Remaining 15 Items Developmental Sample 1 Developmental Sample 2 Developmental Sample 3 Validation Sample

16. Decision Tree Example Overall 85.9% n=11,286 Arts & Sciences, Business Admin., Hospitality Mgmt. Health & Pub. Affairs Engineering Education 85.8% n=6,460 72.7% n=378 91.5% n=2,079 86.7% n=2,369 F2F, E, M W F2F E, M, W F2F E, M 89.1% n=1,043 79.6% n=230 94.1% n=1,036 64.7% n=148 74.8% n=821 86.5% n=5,639 females males A&S BA & Hosp. mgmt 84.1% n=2,376 78.5% n=526 88.4% n=3,263 68.9% n=298

17. Rule #1: If... Excellent Very Good Good Fair Poor Facilitation of learning & Communication of ideas Then... The probability of an overall rating of Excellent = .93 & The probability of an overall rating of Fair or Poor =.00

18. A comparison of excellent ratings by college unadjusted and adjusted for instructors satisfying Rule 1 College Unadjusted % Adjusted % Arts & Sciences41.6 92.4 Business 34.9 90.9 Education 56.8 94.8 Engineering 36.2 91.3 H&PA 46.1 93.9 (N=441,758) (N=147,544)

19. A comparison of excellent ratings by course modality--unadjusted and adjusted for instructors satisfying Rule 1 Course Modality Unadjusted % Adjusted % F2F 42.0 92.2 E 44.0 92.3 M 40.6 92.0 W 55.4 92.7 ITV 20.9 86.7 N=709,285 N=235,745

20. Rule #2: If... Excellent Very Good Good Fair Poor Instructor is interested in your learning Organization of the course Respect & concern for students Communication of ideas Facilitation of learning Then... The probability of an overall rating ofExcellent = .84 & The probability of an overall rating ofFair or Poor=.00

21. A comparison of excellent ratings by college unadjusted and adjusted for instructors satisfying Rule 2 College Unadjusted % Adjusted % Arts & Sciences41.6 86.8 Business 34.9 81.6 Education 56.8 81.9 Engineering 36.2 84.4 H&PA 46.1 84.5 N=441,758 N=15,207

22. A comparison of excellent ratings by course modality--unadjusted and adjusted for instructors satisfying Rule 2 Course Modality Unadjusted % Adjusted % F2F 42.0 84.3 E 44.0 84.6 M 40.6 85.6 W 55.4 87.5 ITV 20.9 71.6 N=709,285 N=235,745

23. Rule #3: If... Excellent Very Good Good Fair Poor Instructor is interested in your learning Respect and concern for students Expresses expectations for student performance Communication of ideas Organization of the course Facilitation of learning Use of class time Then... The probability of an overall rating ofExcellent = .78 & The probability of an overall rating ofFair or Poor=.00

24. A comparison of excellent ratings by college unadjusted and adjusted for instructors satisfying Rule 3 College Unadjusted % Adjusted % Arts & Sciences41.6 81.6 Business 34.9 78.0 Education 56.8 74.5 Engineering 36.2 80.0 H&PA 46.1 77.6 N=441,758 N=15,060

25. A comparison of excellent ratings by course modality--unadjusted and adjusted for instructors satisfying Rule 3 Course Modality Unadjusted Adjusted F2F 42.0 83.9 E 44.0 86.5 M 40.6 84.0 W 55.4 88.4 ITV 20.9 81.1 N=709,285 N=235,745

26. A conceptual path diagram for an instructor receiving an overall rating of Excellent at UCF • Facilitate student learning .93* Excellent Rating Facilitative climate • Interested in student learning • Well organized course • Respect and concern for students Prerequisite Ability to communicate info. and ideas .84* • Clear expectations for students • Respect & concern for students • Interested in student learning • Effective use of class time • Well-organized course Excellent Rating .78* • Supportive environment • Organization * probability of an excellent rating