Milwaukee Mathematics Partnership

Milwaukee Mathematics Partnership Program Evaluation Year 4 Results Cindy Walker University of Wisconsin-Milwaukee Carl Hanssen Hanssen Consulting, LLC

Evaluation Goals • Help the MMP better serve its constituents and improve its effectiveness • Serve the broader mathematics education community through documentation and dissemination of MMP activities

Student Achievement Distal Outcomes Classroom Practice Teacher Content & Pedagogical Knowledge Proximal Outcomes Teacher Involvement Learning Team Effort School Buy-in MMP Activities New Courses Math Faculty Involvement District Buy-in UWM Buy-In MATC Buy-In MPA Ownership MMP Evaluation Logic Model

Student Achievement Classroom Practice Teacher Content & Pedagogical Knowledge Teacher Involvement Collaboration Learning Team Effort School Buy-in Focus of MMP Evaluation

Presentation Overview • District Trends • Student Achievement • Case Study Analyses

District Trends These data show that there has been progress toward embracing MMP principles in schools

District Trends:Online Survey Responses Responses are aggregated within a school so that each school counts as 1 case in the analysis

District Trends:Online Survey Variables 75+ Survey items 17 Composite Variables Example Composite Variable: Alignment How aligned a school’s curriculum is to standards and learning targets Items I feel the mathematics program my school uses aligns with: MPS learning targets. Wisconsin state standards. Goals of the Comprehensive Math Framework. State/district assessments (WKCE/Terra Nova).

District Trends: Statistically Significant Improvements Spring 2006 Spring 2007 3.01 2.84 Quantity of PD Consistency in math instruction Engaging in activities to align curriculum to learning targets Engaging in activities using CABS and student work samples Engaging in activities to gauge student progress Talking about teaching & learning of mathematics with others 2.88 3.06 3.17 3.42 3.25 3.60 Engagement 2.88 3.17 2.99 3.72

School Math Focus Consistent curriculum + Teachers working together + PD perceived as valuable Strong Math Focus Predicts

Supportive Learning Teams MTL perceived as supportive + Curriculum aligned to targets + Learning Team focused on mathematics + Teachers working together + PD perceived as valuable Supportive Learning Teams Predicts

Supportive MTL PD perceived as valuable + MTS perceived as supportive + Teachers working together + Learning Team focuses on math - Less engaged in activities designed to align curriculum to targets Supportive MTLs Predicts Schools with a supportive MTL likely aligned curriculum to targets last year

1. Conclusions • Across the district, schools are reporting higher levels of involvement with MMP and more frequently engaging in activities that the MMP encourages and promotes • There seems to be an increase in the number of school staff who are responsible for helping the school focus on improving mathematics teaching and learning • Dispersed throughout schools with quality LTs and MTLs

2. Student Achievement Are student achievement gains greater in schools that have more fully embraced MMP principles?

Analytical Approach: HLM Use Student Achievement Data from 2005 + MMP Online Survey Results from 2006 to explain variability inStudent Achievement in 2006

Sources of Variability in Student Achievement Scores 12% MMP Alignment 4th Grade 9% LT Quality 19% School 79% Other Variability In Student Achievement In 2006 52% Student Achievement In 2005 81% Student 48% Other

Sources of Variability in Student Achievement Scores 7th Grade 10% MMP Alignment 24% School 90% Other Variability In Student Achievement In 2006 58% Student Achievement in 2005 76% Student 42% Other

HLM Residual Analysis

Student Achievement: Conclusions • Schools that report having greater alignment between math curriculum and learning targets are more likely to attain higher student achievement gains in mathematics • Learning team influence, in terms of increasing student achievement in mathematics, seems to be greatest in the lower grades

HLM Analysis: Future Directions • Longitudinal analyses to determine trends over time • School effectiveness ranking likely dependent on HLM modeling approach used • Determine which modeling approach (i.e. Difference model, Growth model, 2 vs. 3-level model) makes most sense by using additional information provided from case study schools • Compare models with and without covariates

Ten Case Study Schools • Diverse set of schools • School Type • 5 K-5 • 3 K-8 • 2 6-8 • Geography • 7 North • 3 South • Median students = 430

Case Study Data Collection • 20 learning team observations—2 in each school • 40 classroom observations—4 in each school; 2 teachers observed 2 times each • MKT Assessment for math teachers • SNA Survey for mathteachers and administrators

Results of Learning Team Observations Team Functioning Leadership Participation Organization/Structure Results Overall Functioning MMP Issues Math Vision Consistency Math Leadership MMP Work Overall MMP Strengths Participation Organization/Structure Strengths Math Leadership Vision, Consistency Areas to Improve Meeting Results Areas to Improve MMP Work

Results of Learning Team Observations Authoritarian Directive leader Little discussion Reporting out Participatory Active discussion Consensus building Planning Key Observation: to what degree are LTmeetings about learning versus schooladministration?

Focus on learning Distributed leadership Positional authority is less important Multiple views are represented and heard Multiple segments of the school are represented Written agenda, note taker, facilitator Explicit action items Participants have hi knowledge and skill levels Focus on administration Principal does all the talking A few individuals dominate the discussion No agenda or team is easily distracted from the agenda Little follow-through on assignments No clear action items Characteristics of High &Low Rated Learning Teams—Team Functioning High Low

Consistent curriculum Math is addressed alongside and in combination with other subjects Coherent within grades and across grades MTL clearly in charge with respect to math Attention to CABS; reference to MMP courses; reviewing student work Variation in curriculum Math not addressed at the meeting No clear math leader—i.e., hard to tell who the MTL is Confusion about the MMP and CMF Characteristics of High & Low RatedLearning Teams—MMP Issues High Low

Conclusions: Learning Team Observations • Schools focused on ‘learning’ during learning team meetings are better positioned to demonstrate strong results • While the participatory approach may be preferred, some schools may need directive leadershipas they work to improve

Results of Classroom Observations General Practice Identify the Math Task Is the Math Correct? Formative Assessment Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Strengths Identify the math task Correct Mathematics Strengths Understanding Reasoning Areas to Improve Formative assessment* Areas to Improve Application Engagement

Characteristics of Strong & Weak Rated Classroom—General Practice Strong Weak • Math is correct • Math task within the lesson was easy to identify • Math task was discrete and level-appropriate • Encouraging self-assessment and peer-assessment • Establish criteria for proficiency • Promoting problem solving and independent thinking • Incorrect Math • Math task was too complex or obscure • Only feedback provided was if answer was correct • Little teacher involvement in the lesson • Feedback focuses on student behavior

Characteristics of Strong & Weak Rated Classroom Performance—CMF Strong Weak • Student explanations sought • Computation is presented as a means to an end • Problem solving was emphasized • Students had to justify solutions • Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives • Close ended questions are emphasized • Only one way to solve problems presented • Minimal time allowed to share solutions • Students not accountable for responding to questions • Problems not presentedin context

Social Network Analysis • Teachers and administrators in each school were asked to name individuals with whom they communicated about mathematics • This is a key indicator of distributed leadership

Results of Social Network Analysis High Low Loose Network MTL Not Central Few Links to MTL MTS Outside Few Links to MTS Tight Network MTL Central Many Links to MTL MTS Inside Many Links to MTS

Low Student Achievement: 2006: 20% Proficient 4-year trend: -4%

High Student Achievement: 2006: 50% Proficient 4-year trend: +7%

Conclusions: Social Network Analysis • The MTL and MTS network positions are good indicators of MMP impact within school-based networks • Distributed leadership really begins to take hold when teacher communication networks aretightly webbed

Overall Conclusions • There is support for the argument that schools that have more fully adopted MMP principles are demonstrating stronger outcomes—though there is still a lot of work to do. • No single factor—e.g., alignment, distributed leadership or learning team performance—is sufficient for success, but all may be necessary

Overall Conclusions • Schools that are performing well do many of the things MMP promotes well, andrealize synergy between many of these activities and principles • MMP impact, though, is not being felt in all schools—thereis tremendous variability in MMPadoption and progress across the district

Future Considerations • Important considerations for sustaining MMP work • Creating Distributed Leadership in a school takes time—and communication is criticalLast year the Learning Team was perceived as the most important actor for improving mathematics teaching and learning.This year, in schools that report high levels of math focus, that responsibilityseems to be dispersed throughout the school.

Future Considerations • Important considerations for sustaining MMP work • MTL role may be shifting from focal point to facilitator—we see growth in the number of staff primarily responsible for helping the school focus on improving mathematics teaching and learning • MTS role may be more importantthan ever—schools using the MTSappear further down the path

Milwaukee Mathematics Partnership