With fiscal pressures to produce many students can educational leadership programs achieve quality?

1. With fiscal pressures to produce many students can educational leadership programs achieve quality? Mark J. Fenster fenster_21stcentury@hotmail.com Valdosta State University

2. Fiscal pressures paper Paper presented at the annual meeting of the Southeastern Evaluation Association Tallahassee FL February 2, 2006 HaPpY gRoUnDhOgDaY!

3. Introduction • Arthur Levine wrote and disseminated a critique of administrator preparation in the United States, “Educating School Leaders”. The study was widely distributed in the popular press and received national and international attention.

4. Introduction • The American Educational Research Association (AERA) Division A put out a special newsletter in the spring 2005 to address some concerns of the Levine (2005) study. • Young, Grow, Orr, and Ogawa (2005) point out in their rebuttal of the Levine study (p. 4), “for example, he [Levine] claims that programs [educational leadership programs] have low admissions standards but provides as evidence the mean scores of test takers, not those who were admitted”.

5. Introduction • If students desiring entry into educational leadership programs have low test scores, but those matriculating into these same leadership programs have test scores similar to other graduate programs, then the Levine criticism of low admission standards can be dismissed.

6. Introduction • However, if students matriculating into educational leadership programs have lower test scores when compared to other graduate programs, then the Levine criticism of low admission standards can be supported.

7. Introduction • Using administrative data from one institution, called XYZ, we can test the Levine contention that students matriculating in educational leadership programs have lower test scores when compared to other graduate programs run by the same institution.

8. Leadership preparation • Different paths to certification by prospective educational leaders • university training program to receive an advanced degree and certification. • Alternative certification programs for people with managerial experience from business have sprung up in some states. • Additionally, some very large school districts (like New York City) run their own leadership preparation program (Gootman, 2004).

9. Leadership preparation • The New York City leadership program is particularly interesting because the goal in the program is to teach people how to become principals.

10. Leadership preparation • As Levine (2005) notes, nobody knows if university sponsored educational leadership programs are any better than a district based program, or one of the alternative programs in training effective educational leaders. Without a base of knowledge to say that one type of leadership preparation program is better than another, a mix of potential avenues to achieve leadership certification can be expected to continue to exist in the next few years.

11. What to do with the Ed.D. degree? • What to do with the Ed.D. degree in educational administration • Levine (2005) also focused attention on the doctorate in educational administration, especially the doctorate of education degree (Ed.D). Levine argued to do away with the Ed. D. degree, arguing “the professoriate was cited for poorly preparing their students as researchers, and being inexperienced in or incapable of carrying out or supervising quality research themselves” (p. 44).

12. What to do with the Ed.D. degree? • Hawley (1988) noted that “Few persons teaching in {Ed. Adm.] doctoral programs are now or ever have been involved in research and are not qualified to supervise research. Thus, very little good research is being conducted by faculty and students” (p. 85). • If educational administration professors do little research, is it surprising that their work and their student’s work add little to theory and to the educational administration knowledge base (Achilles, 1990, 1991; Achilles & Finn, 2002)?

13. Methods • Data Sources • The primary data source was the institution’s administrative database, including grade point averages and Scholastic Aptitude Test (SAT) scores (total, verbal and math) on all college of education graduate students. • Additional data came from questionnaires distributed to students by the instructor on their first day of class. This questionnaire asked whether the student had any interest in becoming an educational leader and the reason for enrolling in the program.

14. Methods • Variables • Dependent Variables. • There were three dependent variables analyzed in this study; • (1) SAT scores of graduate students matriculating at XYZ, • (2) grade point averages of graduate students enrolled at XYZ, and • (3) a combined z-score of overall academic performance when enrolled at XYZ.

15. Methods • Independent Variables. • There were three independent variables analyzed in this study. • One independent variable was the program a student matriculated into (psychology, educational leadership, early childhood, middle grades, or music education, to mention some pertinent examples).

16. Methods • A second independent variable was the student’s response to a question whether they had an interest in becoming an educational leader. Data on this variable was collected though a questionnaire handed out to students on the first day of class. • A third independent variable was whether a student became an educational leader. Data on this variable was collected through contacts with former students, and with these contacts we were able to ascertain the exact number of students who became leaders.

17. Results • Analyzing data from XYZ, we found that educational leadership students had the second lowest SAT total scores across 10 graduate programs.

18. Results • Educational leadership students scored, on average, about 84 points below the overall graduate mean on their total SAT scores. • Educational leadership students scored 54 points below the average SAT verbal score, and 30 points below the average SAT math score.

19. Table 1 SAT scores by graduate program

22. Results • Educational Leadership students had one of the lowest average SAT scores, but that did not prevent them from getting virtually all A’s in coursework, as shown in Table 2.

23. Results • Educational leadership students had nearly the highest grade point averages of the ten graduate programs. • The median grade point average of educational leadership graduate students was 3.94, second only to Music Education’s 4.00. • However, students in Music Education had an average SAT score 183 points higher than the educational leadership students.

24. Results • The only program admitting students with lower SAT scores than Educational Leadership was Physical Education. In contrast, students in the Physical Education program had the lowest median grade point average of the graduate programs at XYZ.

25. Table 2 Analysis of graduate student grade point averages by degree program, as of March 1, 2005.

26. Results • Interest in becoming an educational leader • The questionnaire allowed us to more closely analyze those students desiring to become educational leaders compared to those students who reported to us that they had no interest in becoming leaders. At XYZ, about 85% of enrolled students in educational leadership programs reported on a questionnaire that they had no interest in becoming an educational leader.

27. Results • Analysis of our data revealed that those students reporting no interest in becoming an educational leader performed slightly better in the program than students reporting that they wanted to become educational leaders, as shown in Figure 1.

28. Figure 1. Differences in performance between students interested and not interested in becoming educational leaders.

29. Results • Eighty-six percent reported they had no interest in becoming educational leaders. These students averaged about 0.09 standard deviations better than the typical student in academic work. • The students reporting that they desired to become educational leaders (14%) averaged 0.06 standard deviations worse than the typical student.

30. Results • The difference in performance between students with no interest in educational leadership and those who were interested in educational leadership, about 0.15 standard deviation units, was statistically insignificant (t=0.30, df=17, ns). The effect size, 0.16, would be considered below small by Cohen’s (1988) criterion.

31. Results • In Figure 2 we disaggregate students desiring to become educational leaders into two groups; those who were interested in becoming educational leaders but have not yet secured a leadership position, and those who were interested in becoming educational leaders and actually became educational leaders.

32. Results • Those interested in becoming educational leaders, but did not yet become a leader performed well in the program. • These students averaged 0.64 standard deviations better than the typical student.

33. Results • Those who were interested in becoming educational leaders and actually became educational leaders performed poorly in the program. These students averaged 0.75 standard deviations worse than the typical student.

34. Results • These two groups were compared to those students reporting they had no interest in becoming educational leaders. • From Figure 1 we know that students with no interest in becoming educational leaders averaged a performance nearly 0.09 standard deviations better than the typical student

35. Results • The differences presented in Table 4 were statistically insignificant (f=1.22, df=2, ns), with a small effect size (.08) calculated per Cohen (1988, pp. 407-415).

36. Figure 2. Differences in academic performance between those with no interest in educational leadership, those who want to become an educational leader but have not yet secured a leadership position, and those who want to become an educational leader and actually became an educational leader.

37. Discussion • Overall, this study supports one contention of Levine’s (2005) study and refutes one claim by Young et al. (2005). • Specifically, the educational leadership programs at XYZ took in students with nearly the lowest test scores.

38. Discussion • Paradoxically, once in the program, virtually all educational leadership students received the grade of “A” in every course.

39. Discussion • Educational leadership programs have to be able to play the role of gatekeeper if they are going to be a positive force in producing the next generation of effective educational leaders. The practice of letting in weak students, giving all students the grade of “A”, and passing all students through the system meant that all students, both weak and strong were certified to become leaders.

40. Discussion • Superintendents and other personnel in these districts had a plethora of choices when it time to fill leadership vacancies, and given the option of choosing academically weak or strong students, local school districts chose the weakest students to become educational leaders.

41. Discussion • Local school districts may not have known (or cared) that their selected candidates to become leaders from XYZ were from the bottom 15% of their respective classes. • However, if the educational leadership program at XYZ had served a better role as gate-keeper, with more rigor in course grading resulting in fewer “A”s, weaker students may have been filtered out and not able to secure certifications as leaders.

42. Discussion • Leadership preparation programs like XYZ make it easier for local school districts to hire academically weak, marginally qualified people as principals. • These weak students selected as leaders are minimally qualified by the standards of the State and the educational leadership program. • More academically highly qualified people who want to become principals wait for leadership appointments as weak students are chosen first.

43. Discussion • Differences in incentives between university and district based leadership programs • University based educational leadership programs may have greater problems playing the gatekeeper role than large district based programs because university programs generate revenue by producing students. • The incentives in university based educational leadership programs are weighted to succumbing to the all too easy practice of letting in weak students, and then giving virtually every student the grade of “A”.

44. Discussion • The result of this process gives leadership programs accolades within the institution from high ranking administrators due to a program’s high retention and low attrition. • The retention rate of educational leadership students at XYZ routinely exceeded 95%.

45. Discussion • In large district programs, like the New York City example, there would seem to be the greater potential for appropriate gate-keeping, since the program’s reputation would be harmed if graduates consistently performed poorly on the job. • The retention rate of educational leadership students in the first graduating class at the New York City program was 85%. The 15% dropout rate was one piece of evidence cited for the program’s rigor (Gootman, 2004).

46. Discussion • Additionally, the motivation for enrolling in an educational leadership program would be quite different in a district based leadership program, compared to a university based leadership program. • In a district leadership program like the New York City example, 100% of attendees become principals (Gootman, 2004). • However, becoming a principal is not the primary motivation of the vast majority of students entering a university based educational leadership program (Davis et al., 2005; Levine, 2005).

47. Discussion • The primary motivation for students enrolling in a master’s degree in educational leadership is receiving a salary increase (Davis et al.; Levine). The State of Georgia pays students roughly $6400 for people employed in the k-12 environment receiving a master’s degree, irrespective of whether the master’s degree is in or out of field.

48. Discussion • XYZ’s educational leadership programs enroll many of these $6400 desiring salary increasing students because the program is convenient for them to attend, and perhaps, because it is an easy program to complete. • Low admission standards and all “A” grading distributions did not negatively impact the department’s reputation externally because so few of its graduates ever become educational leaders.

49. Discussion • With few graduates as leaders, there are very few “disaster” stories of alumni performing poorly on the job. Stated another way, the 14% who desire educational leadership positions at XYZ were dispersed across many school districts. Some of these 14% may go on to become educational leaders.

50. Discussion • However, the small numbers of people who actually become educational leaders, combined with the relatively large number of school districts in the area (44 in XYZ’s service area), make it unlikely that any one school district would experience multiple job performance failures from graduates of XYZ.