Milwaukee Mathematics Partnership

1. Milwaukee Mathematics Partnership Program Evaluation Year 4 Results Carl Hanssen Hanssen Consulting, LLC Cindy Walker University of Wisconsin-Milwaukee November 2007 MTL Meeting • This material is based upon work supported by the National Science Foundation under Grant No. 0314898. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

2. Evaluation Goals • Help the MMP better serve its constituents and improve its effectiveness • Serve the broader mathematics education community through documentation and dissemination of MMP activities

3. Student Achievement Distal Outcomes Classroom Practice Teacher Content & Pedagogical Knowledge Proximal Outcomes Teacher Involvement Learning Team Effort School Buy-in MMP Activities New Courses Math Faculty Involvement District Buy-in UWM Buy-In MATC Buy-In MPA Ownership MMP Evaluation Logic Model

4. Presentation Overview • 2007 MMP Online Survey Results & Trends • MMP Impact on Student Achievement Gains • Detailed Work in 10 MPSSchools

5. Classroom Practice Teacher Involvement Learning Team Effort School Buy-in 1. 2007 MMP Online Survey Results & Trends

6. Online Survey Responses Responses are aggregated within a school so that each school counts as 1 case in the analysis

7. Online Survey Variables 75+ Survey items 17 Composite Variables Example Composite Variable: Alignment How aligned a school’s curriculum is to standards and learning targets Items I feel the mathematics program my school uses aligns with: MPS learning targets. Wisconsin state standards. Goals of the Comprehensive Math Framework. State/district assessments (WKCE/Terra Nova).

8. Context for these results These data show that there has been progress toward embracing MMP principles in schools

9. Statistically Significant Improvements Spring 2006 Spring 2007 3.01 2.84 Quantity of PD Consistency in math instruction Engaging in activities to align curriculum to learning targets Engaging in activities using CABS and student work samples Engaging in activities to gauge student progress Talking about teaching & learning Mathematics with others 2.88 3.06 3.17 3.42 3.25 3.60 Engagement 2.88 3.17 2.99 3.72

10. School Math Focus Consistent curriculum + Teachers working together + PD perceived as valuable Strong Math Focus Predicts

11. Supportive Learning Teams MTL perceived as supportive + Curriculum aligned to targets + Learning Team focuses on math + Teachers working together + PD perceived as valuable Perception of a supportive Learning Team Predicts

12. Supportive MTL PD perceived as valuable + MTS perceived as supportive + Teachers working together + Learning Team focuses on math - Curriculum aligned to targets Perception of a supportive MTL Predicts Schools with a supportive MTL likely aligned curriculum to targets last year

13. 1. Conclusion • Across the district, schools are reporting higher levels of involvement with MMP • Similarly, schools report more frequently engaging in activities that the MMP encourages and promotes

14. 2. MMP Impact on Student Achievement Student Achievement Classroom Practice Teacher Involvement Learning Team Effort School Buy-in

15. MMP Impact on 2006 Student Achievement Are student achievement gains greater in schools that have more fully embraced MMP principles?

16. Sep 04 State Test Fall 2004 2004-2005 School Year MMP Online Survey Spring 2005 Sep 05 State Test Fall 2005 2005-2006 School Year MMP Online Survey Spring 2006 Sep 06 State Test Fall 2006 2006-2007 School Year MMP Online Survey Spring 2007 Sep 07 State Test Fall 2007 2007-2008 School Year MMP Online Survey Spring 2008 Sep 08 Data Collection Timeline

17. Analytical Approach Use Student Achievement Data from 2005 + MMP Online Survey Results from 2006 to explain variability inStudent Achievement Gains from 2005 to 2006

18. Sources of Variability in Student Achievement Scores 12% MMP Alignment 4th Grade 9% LT Quality 19% School 79% Other Variability In Student Achievement In 2006 52% Student Achievement In 2005 81% Student 48% Other

19. Sources of Variability in Student Achievement Scores 9% MMP Alignment 5th Grade 5% LT Quality 22% School 79% Other Variability In Student Achievement In 2006 56% Student Achievement in 2005 78% Student 44% Other

20. Sources of Variability in Student Achievement Scores 6th Grade 4% MMP Alignment 22% School 96% Other Variability In Student Achievement in 2006 50% Student Achievement In 2005 78% Student 50% Other

21. Sources of Variability in Student Achievement Scores 7th Grade 10% MMP Alignment 24% School 90% Other Variability In Student Achievement In 2006 58% Student Achievement in 2005 76% Student 42% Other

22. Sources of Variability in Student Achievement Scores 8th Grade 7% MMP Alignment 21% School 93% Other Variability In Student Achievement In 2006 56% Student Achievement in 2005 79% Student 44% Other

23. 2. Conclusion • Schools that more fully embrace MMP principles are more likely to show gains in student achievement • MMP influence is perhaps felt most strongly by students in the lower grades

24. Student Achievement Classroom Practice Teacher Content & Pedagogical Knowledge Teacher Involvement Collaboration Learning Team Effort School Buy-in 3. Detailed Work in 10 MPS Schools

25. Ten Case Study Schools • Diverse set of schools • School Type • 5 K-5 • 3 K-8 • 2 6-8 • Geography • 7 North • 3 South • Median students = 430

26. Case Study Data Collection • 20 learning team observations—2 in each school • 40 classroom observations—4 in each school; 2 teachers observed 2 times each • MKT Assessment for math teachers • SNA Survey for mathteachers and administrators

27. Results of Learning Team Observations Team Functioning Leadership Participation Organization/Structure Results Overall Functioning MMP Issues Math Vision Consistency Math Leadership MMP Work Overall MMP Strengths Participation Organization/Structure Strengths Math Leadership Vision, Consistency Areas to Improve Meeting Results Areas to Improve MMP Work

28. Two Learning Team Models Emerging Authoritarian Directive leader Little discussion Reporting out Participatory Active discussion Consensus building Planning Key Observation: to what degree are LTmeetings about learning versus schooladministration?

29. Focus on learning Distributed leadership Positional authority is less important Multiple views are represented and heard Multiple segments of the school are represented Written agenda, note taker, facilitator Explicit action items Participants have hi knowledge and skill levels Focus on administration Principal does all the talking A few individuals dominate the discussion No agenda or team is easily distracted from the agenda Little follow-through on assignments No clear action items Characteristics of High &Low Rated Learning Teams—Team Functioning High Low

30. Consistent curriculum Math is addressed alongside and in combination with other subjects Coherent within grades and across grades MTL clearly in charge with respect to math Attention to CABS; reference to MMP courses; reviewing student work Variation in curriculum Math not addressed at the meeting No clear math leader—i.e., hard to tell who the MTL is Confusion about the MMP and CMF Characteristics of High & Low RatedLearning Teams—MMP Issues High Low

31. Results of Classroom Observations General Practice Identify the Math Task Is the Math Correct? Formative Assessment Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Strengths Identify the math task Correct Mathematics Strengths Understanding Reasoning Areas to Improve Formative assessment* Areas to Improve Application Engagement

32. Math is correct Math task within the lesson was easy to identify Math task was discrete and level-appropriate Encouraging self-assessment and peer-assessment Establish criteria for proficiency Promoting problem solving and independent thinking Incorrect Math Math task was too complex or obscure Only feedback provided was if answer was correct Little teacher involvement in the lesson Feedback focuses on student behavior Characteristics of Strong & Weak Rated Classroom—General Practice Strong Weak

33. Student explanations sought Computation is presented as a means to an end Problem solving was emphasized Students had to justify solutions Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives Close ended questions are emphasized Only one way to solve problems presented Minimal time allowed to share solutions Students not accountable for responding to questions Problems not presentedin context Characteristics of Strong & Weak Rated Classroom Performance—CMF Strong Weak

34. Results of MKT Assessment 43 item assessment addressed 3 content areas: Number & Operations Algebra Geometry (2006-07 Focus) & & Overall Score

35. Results of MKT Assessment MKT scores can be interpreted like z-scores Results were aggregated within schools There is tremendous variability in the results Geometry was the MMP PD focus in 2006-07

36. Social Network Analysis • Teachers and administrators in each school were asked to name individuals with whom they communicated about mathematics • This is a key indicator of distributed leadership

37. Mathematics Distributed Leadership Continuum High Low Loose Network MTL Not Central Few Links to MTL MTS Outside Few Links to MTS Tight Network MTL Central Many Links to MTL MTS Inside Many Links to MTS

38. Low Student Achievement: 2006: 20% Proficient 4-year trend: -4%

39. Medium Student Achievement: 2006: 21% Proficient 4-year trend: -19%

40. High Student Achievement: 2006: 50% Proficient 4-year trend: +7%

41. 3. Conclusion • No single factor—e.g., distributed leadership, teacher MKT, learning team performance—is sufficient for success, but all may be necessary • Schools that are performing well do many of the things MMP promotes well, andrealize synergy between manyof these activities and principles

42. Overall Conclusions • There is support for the argument that schools that have more fully adopted MMP principles are demonstrating stronger outcomes—though there is still a lot of work to do. • MMP Impact, though, is not being felt in all schools—thereis tremendous variability in MMPadoption and progress across the district

43. Overall Conclusions • Important considerations for sustaining MMP work • Creating Distributed Leadership in a school takes time—and communication is criticalLast year the Learning Team was perceived as the most important actor for improving mathematics teaching and learning.This year, in schools that report high levels of math focus, that responsibilityseems to be dispersed throughout the school.

44. Overall Conclusions • Important considerations for sustaining MMP work • MTL role may be shifting from focal point to facilitator—we see a shift in the perception of who is responsible for helping the school focus on improving mathematics teaching and learning • MTS role may more importantthan ever—schools using the MTSappear further down the path

45. Focus Question • What message will you be taking back about… • Your ongoing work to improve math in your school? • Specific areas where your school canimprove its math education?