Strategies for Incorporating Evaluation into Teacher Scholarship Programs:

1. Strategies for Incorporating Evaluation into Teacher Scholarship Programs: A Case Study of Math for America Los Angeles Teacher Fellows Program Pam Mason Executive Director, MƒA LA Rebecca Eddy President, Cobblestone Applied Research & Evaluation, Inc. www.mathforamerica.org/la

2. Math for America A Partnership of Mathematicians, Business People and Educators

3. The mission of Math for America is to improve mathematics and science education in US public secondary schools by building a corps of outstanding STEM teachers and leaders. Our goal is to support those outstanding STEM teachers already in the classroom and to increase the number of mathematically talented individuals entering teaching.

4. In 2008, The University of Southern California (USC), Claremont Graduate University (CGU) and Harvey Mudd College (HMC) teamed together with MƒA to create Math for America Los Angeles (MƒA LA). 2008 Cohort

5. 2008 - 2012 Cohorts

6. Teaching Fellowship To improve math education in our public schools by attracting, training, and retaining highly qualified secondary school mathematics teachers.

7. Master Teacher Fellowship To provide the support teachers require to answer a critical need and bring about a real, sustained change. 2012 Master Teacher Cohort at PCMI

8. Retaining Teachers • Ongoing classroom observations and instructional support by experienced advisors • Social and network building opportunities • Monthly Professional Development • • Curriculum and Teaching Support • National Board Certification Developing Leaders

9. Cluster Placement Clusters of Fellows at individual schools create effective learning environments for students Hawkins High School

10. Montclair High School Miguel Contreras Learning Complex

11. Alain Leroy Locke Senior High SchoolGreen Dot Public School James Monroe High School

12. Helen Bernstein High School Fairfax High School

13. Los Altos High School East Los Angeles Performing Arts Academy

14. Ramon C. CortinesHigh School

15. ProfessionalDevelopment Professional Development Outreach • Program Designed by • Pam Mason, • MƒA LA Executive Director • Darryl Yong, • Associate Chair of • Mathematics at • Harvey Mudd College

16. Relevant, “Just in Time” Professional Development

17. Conferences MƒA LA Fellows have attended regional and national conferences for mathematics teachers and have given formal presentations.

18. Conferences attended during the 2012-2013 school year: • CMC South Conference in Palm Springs • CMC North Conference in Asilomar • NCSM Conference in Denver • NCTM Conference in Denver Photo by Solares Photography Reprinted with permission from the National Council of Teachers of Mathematics

19. The MƒA LA Network A tightly networked community • Regular meetings • Experienced coaches • Social events

20. The MƒA LA Effect … ACCORDING TO MƒA LA COACHES “… a feeling of camaraderie …”

21. MƒA LA Evaluation • Purpose of Evaluation • Setting goals • Identifying indicators to measure program implementation and outcomes • Keeping the program on track • Drawing conclusions about the merit of the program

22. MƒA LA Evaluation • Evaluation designed to answer key evaluation questions • Uses a combination of qualitative and quantitative methods • Quasi-experimental design to compare students of MfA LA Fellows vs. students on non-Fellows in the same schools

23. Key Evaluation Questions

24. MƒA LA Evaluation Math for America Los Angeles is committed to a comprehensive evaluation of its programs and operations • Purpose • Track program implementation and measure outcomes • Design • Mixed method: Quantitative and qualitative data sources to answer evaluation questions • Matched control group quasi-experimental design for student achievement data • Evaluation Activities • Focus groups, interviews, classroom observations, document analysis, assessment of student test scores & surveys

25. Logic Model • What is a logic model? • Common elements in a logic model: • Inputs • Activities • Outputs • Outcomes “Logic models help us plan, implement, evaluate, and communicate more effectively.” Taylor-Powell, E., & Henert, E. (2008)

26. Logic Model Resources • University of Wisconsin – Extension: templates, examples http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html • W. K. Kellogg Foundation: Development Guide http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx

27. Logic Model: Simplest Form HEADACHE Get Pills Take Pills Feel Better INPUTS OUTPUTS OUTCOMES Situation

28. MfA LA Teacher Fellows Program Logic Model Situation Students under-achieve in math; U.S. needs STEM workforce to be globally competitive Outputs Inputs What is invested in MfA LA Outcomes Activities What MfA LA does Short-term Med-term Long-term Participation Who MfA LA reaches Fellows have improved pedagogical skills Fellows retained in math profession Fellows continue to teach in schools Fellows receive training Fellows receive stipends + tuition support Staff Exec. Director Coaches Financial Support (Grants, Donors) Time Cred. Program Teachers at Fellows’ school collaborate with Fellows Fellows receive coaching from mentors Students acquire strong math skills Fellows maintain content knowledge in math LA schools/ districts have highly qualified math teachers Fellows receive prof. support through PD Students maintain achievement & interest in math/ other STEM fields Students receive math instruction from trained Fellow Fellows feel supported & respected as a teacher LA schools/ districts have high achieving math students Fellows receive moral/ emotional/ instructional support from other Fellows Schools receive trained teachers/ leaders National Board Certification & continued professional growth Fellows are more confident to lead in classroom & in department Students from LA competitive in global economy Districts receive trained teachers Fellows teach in classrooms/ work in math depts • External Factors • Students in high poverty schools in LA less likely to receive instruction from highly qualified teachers • Students in high poverty schools in LA less likely to perform well in mathematics • Assumptions • Highly qualified math teachers produce high achieving students • Recruiting, training & supporting highly-qualified individuals will produce highly qualified teachers who stay in the profession

29. Outcomes Outputs Need Inputs Activities Participation Product Short-term Long-term What needs are we addressing? Teachers lack support and professional development experiences that prevents recruitment & retention of high-quality mathematics secondary school teachers

30. Outcomes Outputs Need Inputs Activities Participation Product Short-term Long-term What resources do we need for the program? Director/ staff expertise, funding

31. Outcomes Outputs Need Inputs Activities Participation Product Short-term Long-term What activities are planned to address the needs? Who will participate? Fellows attend monthly PD sessions 10 months per year & attend professional conferences; receive coaching support; stipends Monitoring Implementation

32. Outcomes Outputs Need Inputs Activities Participation Product Short-term Long-term What are the short- term and long-term changes we expect to see in participants? Fellows acquire content knowledge, classroom management & planning skills (short-term) Fellows are retained in the profession & are effective secondary math teachers (long-term) Determining Program Merit

33. Example: Baking Cookies • See the following list and determine in which category each applies • For example: Bake cookie dough • Is this a need, activity, outcome, etc.?

34. Using Your Own Example 1. Start with a need 2. Identify one activity (Outputs) • Specify how much of that activity needs to happen to address the need (Activities) • Establish who will be reached by the activity and what is required of their participation (Participation) 3. What resources are required? (Inputs) 4. What is your goal for meeting the needs? (Outcomes)

35. Logic Model Debrief • What are some common needs that you identified? • In thinking about designing an evaluation, what are some indicators that we will track? • Outputs • Outcomes • How can we design an evaluation to effectively address the key evaluation questions and include the indicators for implementation and outcomes?

36. MƒA LA Evaluation Results

37. Rocky Paneno 2009 Cohort “I am very thankful for all that MƒA LA has been done to make me a better teacher.”

38. Meoshe Williams 2009 Cohort “I am grateful for such a wonderful chance to inspire young minds ...”

39. Joan Caraway 2009 Cohort “I always consider it an honor to say that I am a fellow in such a wonderful organization

40. Renewing and expanding our commitment to improve student achievement and mathematics education in the greater Los Angeles community with each new cohort of fellows.