Interaction with and Reflection on the Noyce Program Evaluation Data

1. Interaction with and Reflection on the Noyce Program Evaluation Data Frances Lawrenz Pey-Yan Liou Christina Madsen Christopher Desjardins Allison Kirchhoff University of Minnesota

2. Noyce Program Evaluation Group Our evaluation project has four major components: Preparation of an extensive literature review pertaining to recruitment and retention Thematic synthesis through content analysis of project information Statistical analyses to produce quantitative models of the program development Execution of an overall program evaluation plan through collaboration (participatory approach) with existing projects.

4. Methodology: Data sources Surveys PI online survey (N=66) Scholar online survey (N=555) Disciplinary faculty online survey (N=80) Interviews Scholars in progress (N=12) Districts in progress (N= 17) Monitoring data collected for NSF Self-response bias

5. Methodology: Analyses • Frequencies, cross tabs, means • Open-ended responses categorization • Factor and cluster analyses • Regression and HLM analyses using 3 outcome variables: • Factor score: Commitment to teaching in high needs school (Influence of Noyce) • Would you have become a teacher if you hadn’t received the Noyce scholarship? • Would you have decided to teach in a high need school if you hadn’t participated in the Noyce scholarship program?

6. PI Survey

7. Components of the PI survey Project overview Noyce money Program characteristics and organization Contacts and partners

8. Project overview 75 active Noyce projects with 88% of the projects responding • Of the 9 non-responding PIs, 7 were in their first or second year and subsequently had no data to report • Projects in 29 states responded, with 12 responding from Texas • Total of 141 strands reported with an average 1 or 2 teacher education strands per responding PI • 8 strands was the maximum • Strand breakdown: • 49 Undergraduate program leading to a bachelor’s degree (34.8%) • 17 Teaching credential (no degree) (12.1%) • 27 Post-bac or graduate program (no master’s awarded) (19.1%) • 27 Graduate program (19.1%) • 21 Other (14.9%)

9. Noyce money • 55% indicated that Noyce scholarship paid for over 75% of scholars’ tuition (n = 66)(PI Sec II Q4) • 60% indicated that their teaching preparation program would continue mostly unchanged without the Noyce funding (n = 65)(PI Sec II Q7) • In contrast, 62% reported that the funding made their teacher certification program different (n = 65)(PI Sec II Q8) • In general, PIs indicated that Noyce funding: • Greatly increased their ability to recruit a variety of students • Had a lesser effect on perceptions about teaching careers • Had a lesser effect on relationships with community, districts, STEM faculty, and industry.(PI Sec II Q9)

10. Funded activities Miscellaneous Meetings Mentoring PI Sec II Q 1, number of respondents for each item ranged from 65 to 56.

11. Program characteristics • Required activities(PI Sec III Q2) • Most: Teaching methods classes specific to candidate’s subject area; student teaching experience • Least: Mentoring during the candidate’s second year of teaching • Required field experience(PI Sec III Q3) • Most: Supervised actual classroom teaching in high needs schools (mode # of hrs > 121). • Least: Education field experience working outside of schools with young people like those who attend high needs schools in your area (mode # of hrs 1-40) • Collaborations with school districts(PI Sec III Q5) • Most: School districts providing practicum sites for scholars • Least: The school districts agreeing to hire all scholar who successfully complete the program.

12. Screening criteria for selecting Noyce scholars PI Sec III Q 3

13. Scholar Survey

14. Components of the scholar survey Project overview Program characteristics and organization Teaching environment and experience The decision to become a teacher Background and experience Overall experience

15. Project overview • Of the 555 scholars responding to the survey: • 46% were teaching full-time/part-time • 31% were still in their certification program but not yet a full-time teacher • 13% were still in their program but also teaching full-time • 8% completed a program but never taught • 1% left their program without completing certification • 1% taught after being certified and were working in education but not as teachers • 1% taught after being certified but were no longing working in education

16. Program characteristics • Features of teacher certification program (Scholar Sec II Q1) • Most: Classes in teaching methods specific to their subject area. • Least: Guaranteed job at a participating school district. • Developing specific strategies for teaching students from diverse racial and ethnic backgrounds was the most extensively explored opportunity across the programs (Scholar Sec II Q4)

17. Teaching environment • 83% of the 320 responding scholars are currently teaching in a high needs school (Scholar Sec III Q1) • 68% of the 324 responding scholars are in schools where over 50% of students receive free or reduced lunch (Scholar Sec III Q3) • 41% of the 322 responding scholars are in schools where there has been 15% teacher attrition over the last 3 years (Scholar Sec III Q3) • 78% of the 318 responding scholars have held leadership positions within the last 3 years (Scholar Sec III Q5)

18. Decision to become a teacher • 53% of scholars decided to become a STEM teacher during adulthood (23 or older) (n = 540) (Scholar Sec IV Q1) • 87% of scholars learned about the Noyce scholarship after they decided to become a teacher (n = 543) (Scholar Sec IV Q4) • 61% of scholars learned about the Noyce scholarship from their advisor (n = 586) (Scholar Sec IV Q5)

19. Factors influencing STEM teacher decision Scholar Sec IV Q3, respondents may have multiple selections (respondents vary from 516 to 535)

20. Effect of Noyce scholarship on teaching Would you have become a teacher if you had not received the Noyce Scholarship? Would you have taught in a high needs school if you hadn’t participated in the Noyce program? Yes Possibly No I will not teach in a high needs school Scholar Sec IV Q6 (n = 543) & Q7 (n = 542)

21. Faculty Survey

22. Faculty survey results 91% were familiar with the Noyce program (n = 79) (Fac Q1) 44% interact with STEM education faculty daily and 25% interact weekly (n = 79) (Fac Q2) Half of those interacting daily were in mathematics Faculty encouraged students to pursue teaching because of the students’ interest or because of the students’ personalities (Fac Q6a) Most suggested response from faculty to students interested in teaching was that they should get actual experience teaching and that teaching is a ‘noble’ profession (Fac Q6b) 57% indicated that availability of scholarships affects advice they give to STEM majors (n = 69) (Fac Q6c)

23. Ability of STEM majors who intend to teach compared to those that do not Fac Q4 (n = 80)

24. Perceived effect of Noyce on STEM teacher certification program Fac Q1 (n = 71)

25. Group Work 1

26. Group work 1: Data analysis • Work in small groups or individually for 10 minutes to develop a question that you would like us to explore • At the end of the 10 minutes hand in your question and we’ll select a suggestion(s) to analyze • We’ll discuss these results and implications after our next group work activity

27. PI Suggestions Career/non-career changers Timing of teaching decision Unique program characteristics

28. Career/non career changers Overview Demographics High needs teaching

29. Overview Career changers ½ of all responding scholars indicated they were ‘career changers’ (n=532) Of the scholars indicating being career changers: All scholars at seven institutions indicated they were ‘career changers’ 25 scholars at one institution indicated they were ‘career changers’ Of the scholars indicating not being career changers: All scholars at four institutions indicated they were ‘non-career changers’ 20 scholars at one institution indicated they were ‘non-career changers’

30. Demographics Of the 267 ‘career changing’ scholars: Program type 67% are/were in graduate programs (masters) 14% are/were in teaching credential programs (no degree) Teaching status 47% are/were teaching full-time 5% completed a teacher program, but never taught Of the 265 ‘non career changing’ scholars: Program type 37% are/were in undergraduate programs 9% are/were in teaching credential programs (no degree) Teaching status 43% are/were teaching full-time 10% completed a teacher program, but never taught

31. High needs teaching Career changers Non-career changers Teaching high needs school Teaching another type of school Scholar Sec III Q1 & Sec V Q4, career changers (n=267), non-career changers (n=265)

32. Timing of teaching decision Scholars were asked, “Did you first learn about the Noyce scholarship before or after you decided to become a teacher?” Program type Recruitment strategies

33. Program type Scholar Sec I Q1 & Sec IV Q 4, Before (n=74), After (n=469)

34. Recruitment strategies Scholar Sec IV Q4 & Q5, Before (n=74), After (n=469)

35. Unique program characteristics Experiences in teacher certification program Required field experiences

36. Experiences • Scholars responded to 14 experiences in their teacher certification program(Scholar Sec II Q1) • Of the 14 experiences, 2 were unique: • Guaranteed job at a participating school district • 27% of scholars indicated they were guaranteed a job (n=544) • Of the scholars indicating yes, 38% were from graduate programs (masters) whereas 9% were from teaching credential programs (n=144) • Mentoring provided by certification program during the second year • 32% of scholars indicated they received second year mentoring by their program (n=524) • Of the scholars indicating yes, 38% were from graduate programs (masters) whereas 16% were from teaching credential programs (n=165)

37. Requiredfield experiences • Scholars responded to 5 required field experiences in their teacher certification program(Scholar Sec II Q 2) • Of the 5 field experiences, 2 were unique: • Education field experience working outside of schools (e.g. summer camp) with young people like those who attend high needs schools in the area • 11% of scholars indicated they received education field experience outside of schools (n=541) • 39% of the scholars were in undergraduate program leading to a bachelors degree whereas 12% were from teaching credential programs (n=57) • Research field experience in your subject area • 27% of scholars indicated they received research field experience (n=539) • 44% of scholars were in graduate programs (masters) whereas 11% were in teaching credential programs (n=143)

38. Group Work 2

39. Group Work 2: Discussion Spend a few minutes thinking about the discussion questions included in your packet. Then in a small group, you’ll discuss either the first or second question. We’ll then reconvene and discuss as a large group ideas that were raised during the small group discussion.

40. Inferential Statistics

41. Factor analysis Factor analysis was utilized to discover underlying factors and as a psychometric procedure for the development and refinement of the questionnaire One main goal is to combine many items into a construct, and use this construct to do more analyses

42. Factor analysis results of Noyce scholar survey

43. Commitment to teaching in high needs school (influence of Noyce) • Become a teacher (Scholar Sec IV Q8a) • Complete the certification program (Scholar Sec IV Q8b) • Take a teaching job (Scholar Sec IV Q8c) • Teach in a high needs school (Scholar Sec IV Q8d) • Remain teaching in a high needs school for the full term of your commitment (Scholar Sec IV Q8e) • Remain teaching in a high needs school beyond the full term of your commitment (Scholar Sec IV Q8f)

44. Hierarchical cluster analysis of commitment to teaching in a high needs school 1 2 3 4 5 6 1.teach in a high needs school (Scholar Sec IV Q8d) 2. remain teaching in a high needs school for the full term of your commitment (Scholar Sec IV Q8e) 3. remain teaching in a high needs school beyond the full term of your commitment (Scholar Sec IV Q8f) 4. become a teacher (Scholar Sec IV Q8a) 5. take a teaching job (Scholar Sec IV Q8c) 6. complete the certification program (Scholar Sec IV Q8b)

45. Influence of Noyce vs. timing 1: before 2: after (Scholar Sec IV Q4) **. Correlation is significant at the 0.01 level (2-tailed).

46. Influence of Noyce vs. timing High Commitment to high needs teaching Spearman’s rho = -0.250 Low Before After Did you first learn about the Noyce scholarship before or after you decided to become a teacher? (Scholar Sec IV Q4) * Not the actual relationship, just a visual representation

47. Influence of funding on timing of teaching decision (Scholar Sec IVQ6) Would you have become a teacher if you had not received the Noyce scholarship? (Scholar Sec IVQ4) (2, N=542) = 83.296, p<0.001

48. Hierarchical Linear Models (HLM) • Also known as multi-level analysis, is a more advanced form of simple linear regression and multiple linear regression Why is HLM used in analyzing Noyce data? • The program-level variables help to explain between-program variance rather than within-program variance • Scholars in the same program tend to be more alike than in other projects

49. Three outcome variables in HLM • Factor score: Commitment to teaching in a high needs school (influence of Noyce) • Would you have become a teacher if you hadn’t received the Noyce scholarship? (Scholar Sec IV Q6) • Would you have decided to teach in a high needs school if you hadn’t participated in the Noyce scholarship program? (Scholar Sec IV Q7)

50. Group Work 3