Maximizing Evaluation Impact by Maximizing Methods:

1. Maximizing Evaluation Impact by Maximizing Methods: Social Network Analysis Combined with Traditional Methods for Measuring Collaboration Carl Hanssen, PhD & MaryAnn Durland, PhD American Evaluation Association Baltimore, MD November 7, 2007

2. Agenda • Social Network Analysis: The Method • SNA Results and Interpretation • Next Steps

3. SNA Methodology • Network Analysis is the study of the relationships formed by the interaction or links between components in a “set”. • MMP sets are schools • The components are individuals: • Faculty, both math and non math • MTL (School level Math Teacher Leaders) • MTS (District level Math Teacher Specialists) • Relationship is communication about math

4. Measures • Indegree – popularity • Density – how “thick”, how much, out of potential

5. Role in Evaluation • How much does the communication structure actually fit the theory and the design of the project • Can the structure be correlated with other measures of implementation and impact? • Activities • Proximal measures

6. Student Achievement Distal Outcomes Classroom Practice Teacher Content & Pedagogical Knowledge Proximal Outcomes Teacher Involvement Learning Team Effort School Buy-in MMP Activities New Courses Math Faculty Involvement District Buy-in UWM Buy-In MATC Buy-In MPA Ownership MMP Evaluation Logic Model

7. MMP Report Card Indicators • 19 indicators in 7 domains derived from in-school data collection, online surveys, and MPS data • MTS Assessment • Collaboration • Learning Teams • Classroom Practice • Professional Development • Teacher MKT • Student Achievement

8. SNA In Context: Evaluation Results Overall rating = 3.5 Gap MTL v. other teacher = .2 Teacher Engagement = 3.2 WKCE Mean % Proficient = 44% Student Achievement PD Hrs. = 17.8 Facilitation Hrs. = 1.0 PD Quality = 3.1 Classroom Practice Teacher Content & Pedagogical Knowledge MTS Assessment = 38.3 of 55 Teacher Involvement Overall IRT = -0.34 Algebra IRT = -0.18 Learning Team Effort School Buy-in Team Functioning = 3.5 MMP Principles = 3.6 LT Quality = 3.1 Network density = 6.7% / School density = 17.6% MTL Role = 13.8 / MTS Role = 5.3 SR MTL Engagement = 4.4 / MTS Quality = 3.0

9. Data Collection • Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics • Statistical analysis focused on • Network and in-school density • Importance of MTL and MTS

10. MMP Impact Continuum Low High Tight Web MTL Central Many Links to MTL MTS Inside Many Links to MTS Loose Web MTL Not Central Few Links to MTL MTS Outside Few Links to MTS

11. Low

12. Medium

13. High

14. Student Achievement & In-School Network Density

15. Student Achievement & MTL In Degree

16. Conclusions • Distributed leadership—a key program goal is manifested by a tightly webbed network • School-level adoption of program principals is manifested by positioning of key individuals within the network • There may be a natural evolution of school networks that is indicative of program impact in that school

17. Next Steps • Continue school-level analysis to strengthen our hypothesis about the relationship between social networks and other proximal and distal outcomes • Develop cross-school (or district-wide) networks

18. Contact Information • Carl Hanssen, PhDHanssen Consulting, LLCcarlh@hanssenconsulting.com616-808-2867 • MaryAnn Durland, PhDDurland Consultingmdurland@durlandconsulting.com630-650-9944