Milwaukee Mathematics Partnership

1. Milwaukee Mathematics Partnership Program Evaluation MTL Meeting November 6,7 2006

2. MMP Goals • Comprehensive Math Framework • Distributed Leadership • Teacher Learning Continuum • Student Learning Continuum

3. MMP Core Partners • University of Wisconsin—Milwaukee • Milwaukee Public Schools • Milwaukee Area Technical College

4. Evaluation Goals • Help the MMP better serve its constituents and improve its effectiveness • Serve the broader mathematics education community through documentation and dissemination of MMP activities

5. Student Achievement Distal Outcomes Classroom Practice Teacher Content & Pedagogical Knowledge Proximal Outcomes Teacher Involvement Learning Team Effort School Buy-in MMP Activities New Courses Math Faculty Involvement District Buy-in UWM Buy-In MATC Buy-In MPA Ownership Evaluation Logic Model

6. 2005-06 Evaluation Activities • MMP Online Survey • MTS Survey • Learning Team Observations • Classroom Observations • Assessment of Teacher MKT • Social Network Analysis • MPS Data Mining

7. Presentation Overview • Part I: District Wide Analysis • Part II: School Case Studies

8. Student Achievement Classroom Practice Teacher Involvement Learning Team Effort School Buy-in Part I: District Wide Indicators

9. Part I Activities • MMP Survey • Designed to measure differences in the quantity and quality of MMP related activities • MTS Survey • Designed to measure how well MTS perceived school to be, in terms of meeting the goals of the MMP • MKT Assessment • Designed to assess teachers’ mathematical knowledge for teaching • WKCE Tests • Designed to assess students’ mathematical content knowledge

10. Respondents Number of MPS Respondents by Role in the MMP

11. Research Questions • Validity of MMP Survey • Change in MMP Impact • Characteristics of math-focused schools • MMP Impact on student achievement • Characteristics of quality learning teams • Characteristics of quality MTLs

12. Research Question #2 • Has the perceived impact or focus of the MMP changed since last year? Analytical Approach • Dependent t-tests conducted at the school level for all school level variables obtained in both administrations of the MMP Survey

13. Results – Trends in Impact of MMP • This year statistically significant increases • MTLs reported discussing mathematics with others at their school (t(90) = 12.06, p < .001) • Teachers reported engaging in activities designed to align their curriculum to standards (t(111) = 8.53, p < .001) • Teachers reported engaging in activities designed around CABS or student work samples (t(106) = 7.04, p < .001)

14. Research Question #3 • What variables characterize schools that are more focused on increasing student achievement in mathematics? Analytical Approach • Stepwise multiple regression

15. Results – Characteristics of Schools with Greater Math Focus • 68% of variability in schools’ overall self-reported focus on mathematics could be explained by differences in: • Teachers reported working together to improve content and pedagogical knowledge (b = .46, t = 6.7, p < .001) • Teachers reported consistent instructional practices used at their school (b = .14, t = 2.4, p = .018) • Teachers perceived the Learning Team to be supportive of efforts to improve math teaching and learning (b = .38, t = 5.6, p <.001)

16. Research Question #4 • What variables help to explain differences in the percentage of students classified as proficient in mathematics? Analytical Approach • Stepwise multiple regressionscontrolling for previousachievement and SES

17. Results – Impact of MMP on Increasing Student Achievement • Schools with a stronger focus on increasing student achievement in mathematics are have a higher percentage of students proficient in mathematics, after controlling for SES and previous achievement (b = .26, t = 3.7, p =.001) • An additional 7% of variability in student proficiency rates explained by the addition of this predictor

18. Research Question #5 • What variables characterize Learning Teams that are perceived to be more helpful in terms of increasing student achievement in mathematics? Analytical Approach • Stepwise multiple regression

19. Results – Characteristics of Supportive Learning Teams • 64% of variability in schools’ overall perception of the level of support provided by the Learning Team could be explained by differences in: • Teachers reported working together on improvement activities designed around CABS or student work samples (b = .41, t = 5.5, p < .001) • Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b = .18, t = 2.4, p = .021) • Teachers perceived the MTL to be supportive of efforts to improve mathematics teaching and learning (b = .46, t = 5.9, p <.001)

20. Research Question #6 • What variables characterize Math Teacher Leaders that are perceived to be more helpful, in terms of increasing student achievement in mathematics? Analytical Approach • Stepwise multiple regression

21. Results – Characteristics of Supportive Math Teacher Leaders • 42% of variability in schools’ overall perception of the level of support provided by the MTL be explained by differences in: • Teachers reported working together on improvement activities designed around CABS or student work samples (b = .38, t = 4.5, p < .001) • Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b = .26, t = 3.0, p = .004) • MTLs perceived themselves as being supported by others at their school (b = .27, t = 3.2, p = .002)

22. Conclusions • MTSs in general have a strong sense of what is going on with school leadership, but less awareness about activity at the classroom level. • MMP efforts are being felt beyond the learning team and MTL by school staff • MMP activities are helping schools become more focused on increasing student achievement in mathematics • Schools that are more focused have increased the proportion of students proficient in mathematics • Adoption of MMP-related principles is reported to be related to supportive learning teams and to supportive Math Teacher Leaders

23. Student Achievement Classroom Practice Teacher Content & Pedagogical Knowledge Teacher Involvement Collaboration Learning Team Effort School Buy-in Part II: Case Study Schools

24. Eleven Case Study Schools • Schools were diverse in terms of • Type • Geography • Student demographics

25. Case Study Data Collection • 22 learning team observations—2 in each school • 44 classroom observations—4 in each school; 2 teachers observed 2 times each • MKT Assessment for math teachers • SNA Survey for matheducation ‘stakeholders’

26. Results of Learning Team Observations Team Functioning Leadership Participation Organization & Structure Results Overall Functioning MMP Issues Math Vision Integration Math Leadership MMP Work Overall MMP Overall, strongest areas were participation & mathematics leadership Biggest areas for improvement were math vision & results

27. Distributed leadership Positional authority is less important Multiple views are represented and heard Multiple segments of the school are represented Written agenda, note taker, facilitator Explicit action items Participants have hi knowledge and skill levels Principal does all the talking A few individuals dominate the discussion No agenda or team is easily distracted from the agenda Little follow-through on assignments No clear action items Characteristics of Hi-Lo Scoring Learning Teams—Team Functioning Hi Lo

28. Consistent curriculum Math is addressed alongside and in combination with other subjects Coherent within grades and (at times) across grades MTL clearly in charge with respect to math Attention to CABS; reference to MMP courses; reviewing student work Variation in curriculum Math not addressed at the meeting No clear math leader—i.e., hard to tell who the MTL is Confusion about the MMP and CMF Too much non-academic housekeeping School climate isthe priority Characteristics of Hi-Lo Scoring Learning Teams—MMP Issues Hi Lo

29. Results of Classroom Observations General Practice Articulating Math Task Formative Assessment Overall Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Overall, strongest areas were articulating the math task & understanding Biggest areas for improvement were use of formative assessment & engagement

30. Math task within the lesson was easy to identify Math task was discrete and level-appropriate Encouraging self-assessment and peer-assessment Establish criteria for proficiency Promoting problem solving and independent thinking Math task was to complex or obscure Only feedback provided was if answer was correct Little teacher involvement in the lesson Feedback focuses on student behavior Characteristics of Hi-Lo Scoring Classroom Performance—General Hi Lo

31. Student explanations sought Computation is presented as a means to an end Problem solving was emphasized Students had to justify solutions Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives Close ended questions are emphasized Only one way to solve problems presented Minimal time allowed to share solutions Students not accountable for responding to questions Problems not presentedin context Characteristics of Hi-Lo Scoring Classroom Performance—CMF Hi Lo

32. Results of MKT Assessment 43 item assessment addressed 3 content areas: Number & Operations Algebra Geometry & & Overall Score

33. Results of MKT Assessment

34. Social Network Analysis • Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics • Statistical analysis focused on • Network and in-school density • Importance of MTL and MTS

35. Overall SNA Results Density—a perfect score is 100% where everyonenames everyone else In-Degree scores are relative measures

36. Example Network

37. Example Network

38. Report Card Indicators • 19 indicators in 7 domains based on in-school data collection, online surveys, and MPS data • MTS Assessment • Collaboration • Learning Teams • Classroom Practice • Professional Development • Teacher MKT • Student Achievement

39. Report Card Results Overall rating = 3.5 Gap MTL v. other teacher = .2 Teacher Engagement = 3.2 WKCE Mean % Proficient = 44% Student Achievement PD Hrs. = 17.8 Facilitation Hrs. = 1.0 PD Quality = 3.1 Classroom Practice Teacher Content & Pedagogical Knowledge MTS Assessment = 38.3 of 55 Teacher Involvement Overall IRT = -0.34 Algebra IRT = -0.18 Learning Team Effort School Buy-in Team Functioning = 3.5 MMP Principles = 3.6 LT Quality = 3.1 Network density = 6.7% / School density = 17.6% MTL Role = 13.8 / MTS Role = 5.3 SR MTL Engagement = 4.4 / MTS Quality = 3.0

40. Student Achievement & In-School Network Density

41. Student Achievement & Learning Team MMP

42. Student Achievement &Professional Development

43. Conclusions • MMP is advancing concepts, ideas, & principles that can help schools improve student achievement results in math. • Schools that score well with regards to MMP-related metrics have higher student achievement. • Learning team adoption of MMP ideas and dense in-school communication networks are predictors of high student achievement

44. Conclusions • At the same time… • Some MPS schools are lagging behind in terms of adopting MMP ideas. • These schools perform do not score as well on MMP metrics, which is consistent with student achievement results. • We know that other factors—prior year student achievement and SES—are stronger predictors

45. Evaluation Next Steps District Wide Analysis • Continue online survey & data mining • Improve ability to link student and teacher data working with MPS Case Study Schools • Recruit case study schools Nov • Plan observations Nov-Dec • Observations Round 1 Jan-Feb • Observations Round 2 March-April • MKT Assessment May • SNA Survey May