Student Evaluations of Faculty and Academic Success

1. Student Evaluations of Faculty and Academic Success Ryan Cartnal Structural Equation Modeling ED 216F Individual Project University of California, Santa Barbara July 2004

2. 1) Research Question Are student evaluations of faculty associated with academic success among a sample of Cuesta community college students?

3. 2) Data - Observed Variables N=365 course sections representing over 20,000 student evaluations of faculty Analysis is performed at the section level given the requirement of student anonymity Student evaluation instrument consists of 27 questions on which students rate faculty excellence (4-point Likert scale: 4 � �excels�, 3 � �meets criteria�, 2 � �needs improvement�, 1 ��unsatisfactory� Mean prior GPA of students in the evaluated section Composite demographic variable consisting of percent of section underrepresented minority (Hispanic, black, American Indian) and average age of section Mean �GPA� of evaluated section � unweighted by units where A=4, B=3,C=2,D=1,F=0, else missing

4. 3) Data � Type of data In a word, messy... Majority of variables were non-normally distributed (Kolmogorov-Smirnov, p<.05, negatively skewed (skew value > 2*SE of skew, extremely leptokurtic). Attempted a reflected inverse transformation of data, which produced negligible results

5. 4) Limitations Majority of variables were non-normally distributed Level of measurement was ordinal on most variables � therefore Polyserial, Tetrachoric, and Polychoric correlations would have been more appropriate than Pearson�s product-moment Forced to examine aggregated data by section rather than by individual student, thus increasing error variance in the measurement of observed variables

6. 5) Research Methodology � Step 1: Exploratory Factor Analysis Performed exploratory factor analysis (EFA) using the 27 survey questions as measured variables (maximum likelihood extraction based on number of factors, direct-Oblimin rotated solution) 5 factors were selected based both on theoretical underpinnings and RMSEA values in the reasonable range (RMSEA<.08) �

7. Research Methodology - Step 2 Confirmatory Factor Analysis Created 5 sub-scales based on EFA results -Quality of Content -Quality of Presentation -Quality of Organization -Quality of Evaluation -Quality of Student Interaction Performed CFA with two factors maximum likelihood extraction using Direct Oblimin rotation and Delta =.3 for better interpretability

8. Research Methodology - Step 2a Confirmatory Factor Analysis Output

9. 9) Research Methodology Step 3: Five A Priori Structural Equation Models Model 1- Quality of Teaching and Classroom Organization along with Prior GPA and demographics are associated with Academic success Model 2- Prior GPA and demographics are associated with Academic success (pedagogical quality is not� 2 factors still in model, but parameters set to 0) Model 3- Same as Model 1 with the correlation between the two factors set at 1 Model 4-Prior GPA and demographics are associated with Academic success (teaching quality is in the model, but parameter is set to 0) Model 5-Prior GPA and demographics along with teaching quality are associated with Academic success. A posteriori model modifications were performed (for fun), but failed to produce better model fits than the a priori models tested

10. 10) Model 1�

11. 10a) Model 1 Output AMOS Model 1 Output Fit: X2 DF RMSEA NNFI 96.44 15 .122 .878 Assessment: Poor model�.

12. 11) Model 2

13. 11a) Model 2 Output AMOS Model 2 Output Fit: X2 DF RMSEA NNFI 120.098 19 .121 .881 Assessment: Virtually no better than model 2 � poor model�

14. 12) Model 3

15. 12a) Model 3 Output AMOS Model 3 Output Fit: X2 DF RMSEA NNFI 100.671 18 .112 .897 Assessment: Again better, but not by much � poor model fit

16. 13) Model 4

17. 13a) Model 4 Output AMOS Model 4 Output Fit: X2 DF RMSEA NNFI 17.4 9 .051 .976 Assessment: Reasonable to good model fit!!!

18. 14) Model 5

19. 14a) Model 5 Output AMOS MODEL 5 Output Fit: X2 DF RMSEA NNFI 15.4 8 .050 .976 Assessment: Good model fit!!!!

20. 15) Model Fit Comparisons Fit: X2 DF RMSEA NNFI Model 1 96.439 15 .122 .878 Model 2 120.098 19 .121 .881 Model 3 100.671 18 .112 .897 Model 4* 17.4 9 .051 .976 Model 5 15.4 8 .050 .976

21. 16) Possible Interpretations Student ratings of faculty excellence are poor measures of pedagogical effectiveness AND/OR Final grades are poor measures of the degree to which course objectives are met OR If these two variables are appropriate measures of faculty excellence and attainment of course objectives, then it would appear that the two are unrelated, and that student academic history, rather, can explain the majority of variation in academic success.

22. 17) Possible future studies Value added model in which the dependent variable becomes change between pre and post measurement of skills Include peer ratings of faculty excellence along with student ratings Perhaps look at success in the subsequent course