Noyce Evaluation

1. Noyce Evaluation University of Minnesota April 20, 2006 Jim Appleton Marjorie Bullitt Bequette Frances Lawrenz Ann Ooms Deena Wassenberg Technical assistance: David Ernst

2. Overall goals for our project • To contribute to the knowledge base about effective strategies for attracting and retaining high quality STEM teachers • To collaboratively develop a plan to evaluate the Noyce Program that will document overall program accomplishments while celebrating the uniqueness of each project • To conduct the evaluation and disseminate findings in a utility-oriented fashion

3. Our responsibilities • We are: • Collecting and categorizing evaluation plans and instruments • Conducting a comprehensive review of the STEM recruitment and retention literature • Working with ORC MACRO to make effective use of their data • We will: • Work with all the projects to design a program evaluation through virtual and face-to-face meetings • Conduct the evaluation • Disseminate the results in a user-friendly fashion

4. We need you to be effective We need your help to: • Refine our literature data base • Optimize the effectiveness of the evaluation variables and instrument data bases • Plan and conduct the program level evaluation

5. Plan for this two session introductory conference • Showcase our materials and explain how we think they might be useful • Obtain feedback on how to improve • Discuss what might be useful in an overall evaluation of the Noyce Program • Determine the most effective use of the evaluation time at the PI conference

6. Outline of today’s presentation • A tour of the Web site • Demo of the literature data base • Summary of the literature findings • Logic Model • Evaluation variables and instruments • Setting the stage for tomorrow’s discussion

7. Q & A • Questions and Answers about the Introduction

8. A tour of the Web site

19. Q&A • Questions and Answers about the Web site overall

27. Q&A • Questions and Answers about using the Literature Data Base

28. Ongoing review of the R&R literature • Looking for factors that affect recruitment and retention • Chose empirical articles from our database that (based on abstracts) had significant results on factors affecting recruitment and retention • Starting with larger N, quantitative work; integration of other studies will follow • We summarized recent articles and used RAND (2004) summaries of older work • Factors were grouped into larger categories • Our more detailed summary will be posted

29. Adams, 1996 Arnold, Choy, & Bobbitt, 1993 Baker, 1988   Ballou, 1996 Ballou & Podgursky, 1997 Brewer, 1996 Bempah, Kaylen, Osburn, & Birkenholz, 1994 Boe, Bobbitt, Cook, Whitener, & Weber, 1997 Bond, 2001 Carroll, Reichardt, Guarino, & Mejia, 2000 Darling-Hammond, Chung, & Frelow, 2002 Eberhard, Reinhardt-Mondragon, & Stottlemyer, 2000 Galchus, 1994 Hall, Pearson, & Carroll, 1992 Hansen Lien, Cavalluzzo, & Wenger, 2004 Hanushek, Kain, & Rivkin, 2001 Hanushek & Pace, 1995 Henke, Geis, Giambattista, & Knepper, 1996 Henke, Zahn, & Carroll, 2001 Hounshell & Griffin, 1989 Ingersoll, 2001 Ingersoll, 2003 Ingersoll & Kralik, 2004 Jacobson, 1988 Kirby, Berends, Naftel, 1999 Kirby & Grissmer, 1993 Loeb (2000) Lankford, Loeb, Wyckoff, 2002 Marso & Pigge, 1997 Miech & Elder, 1996 Mont & Rees, 1996 Murnane, Singer, Willett, Kemple, & Olsen, 1991 Odell & Ferraro, 1992 Pigge, 1985 Plecki, Elfers, Loeb, Zahir, & Knapp, 2005 Literature examined so far • Rickman & Parker, 1990 • Seyfarth & Bost, 1986 • Shen, 1997 • Shen (Autumn, 1997) • Shen, 1998 • Shen, 1999 • Shin, 1994 • Shin, 1995 • Shugart and Hounsell, 1995   • Stinebrickner, 2001a • Stinebrickner, 2001b • Stinebickner 2002 • Stockard & Lehman, 2004 • Theobald, 1990 Tran, Young, Mathison, & Hahn, 2000 Villar & Strong, 2005 • Weiss, 1999 • Young, Place, Rinehart, Jury, & Baits, 1997

30. What research has shown to affect retention • Characteristics of teachers: • Race/ethnicity • Gender • Experience • Age • Type of training program • Area taught • Academic ability/achievement • Family and fertility choices • Reasons for choosing to teach • Certainty of intention to teach

31. What research has shown to affect retention • Teacher preparation program characteristics • Most of the large N quantitative work that we’ve examined focuses on the type of program (alternative, master’s 5th year, major in education or in a discipline), not program components. • Some studies examine the effects of course requirements generally. • Both program type AND program components matter, though.

32. What research has shown to affect retention • Mentoring and induction programs • Again, details on what helps are underexamined. • Salary • Pay affects retention, interacting with gender, race/ethnicity, other local salaries and conditions, subject taught, potential for advancement (and salaries for those positions), and salary scale/highest salary.

33. What research has shown to affect retention • School and district setting • “School culture” • Race/ethnicity of students; also distribution of race/ethnicity • Student ability • Student SES; also distribution of SES • School size • Number of classes taught • Classes taught in area of specialization • Spending (amount and patterns) • Incidence of crime/violence

34. Putting this all together: • Tracking teacher characteristics (affective as well as demographic), program and mentoring experience (and the connections between those two), salary, district conditions, and more, can help each project improve and can help all projects learn from each other.

35. Q&A • Questions and Answers about what the literature has shown

37. Theoretical Framework: LOGIC MODEL DESCRIPTION Our proposed Noyce Logic Model contains our efforts to delineate several perspectives: • The Noyce Program Ideal • Depicted by the main path as well as bold headings preceded by addition signs (e.g., “+Plan to teach”) • Decision points en route to becoming a STEM teacher • Indicated by diverging routes from the main path describing alternative options and the Noyce Ideal in bold headings • Dashed boxes denote retention/recruitment by school or program • Important STEM major decision factors • Influenced by attributes of the candidate, pre-service program, and school/district (depicted as bulleted lists on the main path) • Depicted as thought bubbles emerging from the decision point

38. Theoretical Framework: LOGIC MODEL DESCRIPTION The Noyce Program Ideal: • Diverse and smart STEM majors will be enticed by scholarships and stipends to enter pre-service programs • Programs will provide adequate and relevant training • These STEM majors will graduate, begin teaching in their field and at high need schools, and fulfill the obligations of their scholarship/stipend. • These new teachers will continue to teach at high-need schools beyond the obligation period?

39. Theoretical Framework: LOGIC MODEL DESCRIPTION Decision Points En Route to Becoming a STEM Teacher: • STEM majors may: • Plan to teach or plan for a non-teaching STEM career • If planning to teach, either enter a certification program or teach without certification • If entering a program, upon graduation decide to teach or to not teach • If choosing to teach, decide if it will be at a low- or high-need school • If at a high-need school, decide whether to remain over time.

40. Theoretical Framework: LOGIC MODEL DESCRIPTION • Important STEM Major Decision Factors along These Paths: • Interests, career values, career pay and importance of monetary compensation, importance of certification, challenge of financial costs, desire and requirement to teach • What value in workplace, social justice beliefs, program/funding requirements, training, fulfillment of job, perception of support and appropriate level of challenge

41. Theoretical Framework: LOGIC MODEL GRAPHIC

43. Q&A • Questions and Answers about the logic model

44. Project Evaluation Resources

48. Project evaluation variables, methods and instruments • Collected from your evaluation plans – Thank You! • Categorize these based on the kind of information collected and how it was collected • Categorize and present any specific evaluation instruments you provided (most now available on our Web site)

49. What you are doing: • Of those responding (41 of 65, 63%), 92.7% of Noyce programs submitted a detailed evaluation plan to us. • We’ve categorized these into: • Evaluation of the program itself • Evaluation of post-program activity • Evaluation methods

50. What you are doing: VARIABLES