Evaluation of the Noyce Teacher Scholarship Program

1. Evaluation of the Noyce Teacher Scholarship Program 2010 NSF Noyce Conference Abt Associates Inc. July 9, 2010

2. Overview of Presentation • Relationship between previous and current studies • What does the national evaluation mean for Noyce awardees and recipients? • Study schedule • Evaluation questions • Approaches to analyses

3. Relationship Between Previous and Current Studies The current study builds on the prior evaluation in several ways: • Updates the evaluation with more current information and encompasses recent program components • Examines the career trajectories of fellows who are beyond their teaching requirement • Includes a more a rigorous impact study to assess program impact on recruitment and retention of STEM teachers in high-need districts and schools

4. What Does the Program Evaluation Mean for Noyce Awardees and Recipients? • This evaluation is a programevaluation, not an evaluation of individual Noyce awardees or recipients. • While data will be collected from individual respondents, all information will be analyzed and reported in an aggregate form to draw conclusions about the impacts of the program as a whole. Individual awardees or recipients will not be identifiable.

5. Role of Noyce Stakeholders • Noyce stakeholders know best about what is happening in their projects, departments, and schools; and Noyce recipients understand how the scholarship and stipends affect their experiences. • Therefore, we hope to work closely with Noyce stakeholders throughout the study, engaging in an ongoing dialogue to guarantee that we obtain the most complete picture of the work of each project. • We will be requesting assistance from PIs in encouraging participants in their projects to participate in surveys and interviews and in ensuring we are interpreting their data correctly.

6. Which Awards will be Included? • The study will initially focus on all Noyce awards funded between 2003 through 2008. • In fall 2010, we will begin surveying all PIs, STEM faculty involved in the Noyce programs, Noyce recipients, and the K-12 principals in which former Noyce recipients are teaching. • In winter 2011, the study will also interview a small subset of these same respondent groups. • In fall 2011, we will survey the same respondents for all 2009 awards, and similarly conduct interviews with a small subset of the respondents. This will enable newer awardees sufficient time to get grant activities started.

7. Planned Data Collection Activities and Schedule Spring 2010: Finalize study design and contact PIs to collect contact information Summer/Fall 2010: Pilot instruments and prepare online surveys Fall 2010/2011: Administer online surveys of PIs, STEM faculty, Noyce recipients, and school principals and collect teacher employment data Winter 2011/2012 Conduct interviews with a sample of Principal Investigators, STEM faculty, and Noyce recipients Spring 2011/2012: Analyze data Fall 2012: Prepare final report

8. Surveys • Types of individuals who participate in the Noyce Program: • Census of current Principal Investigators • Convenience sample ofSTEM Faculty • Census of current recipients • Purposive sample of K-12 Principals • Topics Covered: • Award Information, Recruitment and Selection, Program Activities, Relationships between departments and with High-Need LEAs • Recipients’ Plans for Teaching in High-Need Schools, for Leadership Roles • Respondents’ Perceptions of Program

9. Interviews Purposive sample of types of individuals who participate in the Noyce program: • PIs, to pursue more detailed questions about programmatic decision-making and responsibilities • STEM Faculty, to provide more detailed information on how Noyce programs influence whether and how STEM departments interact with teacher preparation programs • Recipients, to provide more information about programmatic features deemed most and least useful to Noyce recipients, and whether and how Noyce funding have influenced career decisions to date • K-12 Principals, to pursue additional information about how principals perceive qualifications of Noyce teachers and how Noyce has affected the availability of qualified math and science teachers

10. Evaluation Questions • What are the goals and activities of the teacher certification programs from which Noyce grants are housed?How do stakeholders perceive the Noyce Program and Noyce recipients?What are the characteristics of the schools in which Noyce recipients teach? • What are the relationships between characteristics of the Noyce Program, types of Noyce recipients, and recipients’ plans to go into/stay in teaching and leadership roles? • What is the impact of Noyce on teacher recruitment, retention, and teacher effectiveness?

11. Descriptive Analyses • Descriptive analyses will tell us how the Noyce Program is being implemented across all awardees; they will also tell us about how Noyce recipients progress through their teacher preparation and early teaching years. • The descriptive analyses will be complemented by 2 in-depth inquiries: • One focuses on institutional changewithin STEM departments, to learn whether and how STEM faculty take responsibility for teacher preparation • One focuses on supports received by recipients during induction, whether from the Noyce Program and/or the employing school/district, to learn whether and how such support influences decisions to stay in teaching and to enter leadership positions

12. Relational Analyses • Relational analyses will examine variation in whether recipients plans to teach, to teach in high-need districts and schools, in particular, and their leadership activities, as a function of: • Initial status upon program entry (e.g., undergraduate, post-baccalaureate, or STEM professional) • Demographic characteristics of IHEs and recipients • Financial incentives received • School climate

13. Impact Analyses • Impact analyses will test whether receipt of a Noyce award has an impact on an IHE teacher certification program in terms of: • The number of STEM majors/professionals who graduate from teacher certification programs both prior to and post-Noyce • The rate of entry into teaching in high-need schools • The expected retention in high-need schools • Teacher content knowledge at certification • Analyses will be based on certification and employment data collected from state departments of education

14. How Will This Study Help Noyce PIs? • The program evaluation will provide Noyce PIs with information about activities being developed and implemented through other Noyce awards and the program components that are associated with various outcomes for different types of recipients. • Abt Associates staff will present findings from the study at the Annual Noyce conferences.

15. Contact Information For more information about the Noyce Program Evaluation, please contact: • Connie Della-Piana, National Science Foundation, cdellapi@nsf.gov • Ellen Bobronnikov, Abt Associates Inc.,ellen_bobronnikov@abtassoc.com • General Questions: NoyceEval@abtassoc.com