Evaluating Math Recovery: Investigating Tutor Learning

1. Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University

2. Rationale As in many interventions, MR relies heavily on the knowledge and practice of the tutors. A substantial part of what “adopting” MR means for districts is the training of tutors. If tutoring is not effective for students we have to ask the following: Did tutors learn as intended from their training? Did tutors conduct tutoring as intended with students? For the first question, this presentation specifically addresses three key issues: the effect of prior knowledge on tutor learning during training the effect of tutor knowledge on the effectiveness of tutoring change in tutor knowledge over the course of the evaluation study

3. Math Recovery (MR) Tutoring MR is an unscripted intervention that is adapted to each individual student that is tutored. Tutors making ongoing diagnoses of students’ current mathematical thinking/strategies design instruction to fall within each student’s zone of proximal development. Central Tool: MR Learning and Instructional Frameworks Based on 20+ years of research into the development of children's thinking in the area of early number. Learning Framework: Used for diagnosis and divided into stages by content. Instructional Framework: Used to guide tutoring instruction based on a students current stage on the Learning Framework.

4. Tutor Knowledge for MR Math Content Knowledge This is a first grade intervention where the main content involves: number words, counting, numerals, structuring numbers, addition, and subtraction . Math Knowledge for Teaching Math Knowledge for Teaching (MKT; Hill, Schilling & Ball, 2004) is a specialized form of mathematical knowledge specific to how students are likely to solve math problems, which solution methods mathematically generalize, which will be most beneficial for their learning in the long run, etc. Knowledge of the MR Program Specific knowledge the MR Frameworks and Assessments, particularly how to use them to determine students’ current levels and plan tutoring instruction accordingly.

5. Example: 12-9 Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!” Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!” Student 3: “It takes two to get to ten and one more to nine…3!” Math Knowledge for Teaching: Understand which mathematical concepts are involved (backward number word sequence, one-to-one correspondence, relationships between quantities) Anticipate this range of solution strategies and more Understand the relative sophistication of these strategies Knowledge of MR: Naming these strategies, determining a student’s Stage of Early Arithmetic Learning (SEAL) level on the Learning Framework and determining appropriate instruction using the Instructional Framework and other published resources (e.g. Wright, Martland, Stafford, 2006; Wright & Stafford, 2002)

6. Example: 12-9 Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!” Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!” Student 3: “It takes two to get to ten and one more to nine…3!” Taken from Math Recovery Teacher Handbook, 2007

7. Example: 12-9 Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!” Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!” Student 3: “It takes two to get to ten and one more to nine…3!” Taken from Math Recovery Teacher Handbook, 2007

8. Description of Tutors All fully-certified, white teachers: 17 female, 1 male 17 certified for 1st grade (1 for 7th-12th math) Specializations: Mathematics: 3 (two at upper grade levels) Other Subjects: 2 Special Education: 2 Before becoming MR tutors: 10 classroom teachers, 5 Title I teachers Other: 1 Special Education, 1 Instructional Coach, 1 Title I Reading Teaching Experience: General: Range = 3 to 30 years, Median = 11.5 years Mathematics: Range = 2 to 30 years, Median = 7.5 years

9. Tutor Training & Study Sites 18 new tutors were trained at two different regional sites. Training involved a five day summer institute and five additional pull out days in the first months of school. All “fresh” sites in terms of study schools.

10. Tutor Assessments In order to measure the development of the new tutors the project used two measures at the beginning of the study, at the end of year one and the end of year two: Mathematics Knowledge for Teaching Assessment1 The MKT was developed at University of Michigan in order to measure Math Knowledge for Teaching. We used a version of the test specific to elementary number and operations. MR Tutor Knowledge Assessment (TKA) The TKA was developed in coordination with MR developers and experts specifically for this study. It is designed to assess a tutor’s understanding of the MR Frameworks and how to apply them in tutoring and assessment scenarios. Both are multiple choice scenario based assessments intended to put the test-taker into the context of practice 1 (Hill, Schilling, and Ball, 2004)

11. Descriptive Statistics: Pretest Scores *These initial differences between site are statistically significant.

12. Hypotheses H1: MR Tutors who begin with a higher level of Math Knowledge for Teaching will more quickly understand how to use the MR frameworks and assessments. Does MKT pretest score predict TKA pretest score? H2: Tutors who have a higher level of Math Knowledge for Teaching/greater understanding of the MR frameworks will have a more positive effect on student achievement. Does MKT pretest score or TKA pretest score predict variation in treatment effect across tutors? H3: As tutors gain experience tutoring and access to students’ mathematical thinking they will grow in both their Math Knowledge for Teaching and their understanding of the MR frameworks and assessments. Do tutors’ scores on the MKT and TKA increase over time?

13. H1 Theory: MKT and Knowledge of MR Because students’ solution strategies are central to the MR Frameworks, we hypothesize that new tutors who know more about ways students typically solve problems in early number or understand the mathematics involved in solving early arithmetic problems (i.e. have more MKT) would be at an advantage in the initial training in terms of experiences to draw on in attempting to understand and use the MR frameworks and assessments.

14. H1 Method & Results: MKT TKA Linear Regression of MKT pretest scores on TKA scores immediately following training. Y = 0 + 1 (MKT) + e R2 = 0.25

15. H1 : Controlling for Different Training • Site predicts TKA score following training. • The R2 for this model = 0.50 • Site and MKT pretest are significantly correlated (0.57,p=0.01). • Trainers reported that the lower starting point of the teachers at site A led to the differences in training. • Therefore, some of the usefulness of site as a predictor is likely related to the lower MKT that may have led to the differences in training.

16. H2 Theory: Knowledge and Student Outcomes If being an effective MR tutor requires the types of tutor knowledge that we measured then tutors with higher assessment scores should also be the most effective with students. This relationship is mediated by what they do with students during tutoring (fidelity).

17. H2: Pretest Score Effects on Students End of 1st grade and end of 2nd grade treatment-control comparison using previously mentioned controls (e.g. pretest, race, gender, SES). Multi-level model nesting students within tutors and testing the significance of tutor pretest scores (on both MKT and TKA) on predicting the variation of the coefficient on treatment. (Achievement) = 00 + 1(treatment)+2-5 (controls) +  1 = 10 + 11(Tutor Assessment Score) Using all outcomes from the evaluation study: WJIII (Math Fluency, Applied Problems, Quantitative Concepts), MR Proximal, and the MR 1.1 Internal Assessment Given that tutors knowledge may be changing over time we tested this model separately for the end of first grade comparison by year.

18. H2: MKT Pretest on Effect of Treatment • In year 2, at the end of first grade the MKT pretest scores predict the treatment effect as measured by the MR 1.1 Internal Assessment. Treatment Effect Mixed Model *p< 0.1 , ** p <0.05, ***p < 0.01 • There were no effects of MKT on treatment effect as measured by external student assessments. • There was no effect of MKT on treatment effect at the end of second grade.

19. H2: TKA Pretest on Effect of Treatment TKA pretest score is not a significant predictor of the effect of the intervention on any of the student outcomes. We know that the relationship between knowledge and outcomes is likely mediated by practice This points to the importance of future analyses linking knowledge and fidelity for understanding where in the MR model the links are breaking down.

20. H3: MR and Generative Tutor Learning It is a result of the theory and practice of MR tutoring that leads to the hypothesis of generative tutor learning. We would not expect other interventions, different in their core practices, to result in tutors learning in and from their tutoring practice in the same way as with MR. Elements of MR hypothesized to contribute to generative learning: Focus on understanding students’ thinking (Franke, 1998) Records and reflection about the link between student strategies and problems posed by tutors “Space” for tutors to be designers rather than strictly “implementers” (Remillard, 2000)

21. H3: Tutor Learning Hierarchical Linear Growth Model with three repeated measures (on both the MKT and TKA) nested within tutors. The predictor, time, is measured in years and used to predict MKT and TKA. If time is a significant predictor, then teachers improved their scores on these assessments over time, evidence of learning. There is not formal training after the beginning of year one. Ongoing support was limited in general, but this varied a bit based on site. Therefore, systematic growth by the tutors over the two years would indicate that they are learning in and from the practice of tutoring.

22. H3: Tutor Learning On both measures there is evidence of tutor learning over time. Given the prior group differences on these measures, what does including the training site as a predictor on assessment score and as a predictor for rate of learning reveal about differences in learning by training site? *p< 0.1 , ** p <0.05, ***p < 0.01 2This model also controls for the form of the MKT that the teacher took by using a dichotomy indicating when they took form C (either time 2 or time 3, randomly assigned).

23. H3: Tutor Learning on the MKT There is no difference in the rate of learning on the MKT by site. However, there is a significant difference in the average MKT score by site (as previously mentioned). *p< 0.1 , ** p <0.05, ***p < 0.01

24. H3: Tutor Learning on the TKA Possible explanations Different levels of ongoing support led to different learning rates Ceiling effect on the TKA assessment (39 points possible) Teacher learning in practice is curvilinear, where once a certain level of knowledge is reached, either there is little room left to grow OR direct intervention (e.g. more intensive PD) is required to continue to grow *p< 0.1 , ** p <0.05, ***p < 0.01

25. Summary of Findings MKT pretest does predict TKA score after the initial training This suggests that Math Knowledge for Teaching might be one important consideration in choosing tutors who will learn MR quickly. MKT pretest also predicts the effect of treatment in year 2 as measured by the MR 1.1 Suggests the importance of Math Knowledge for Teaching for being an effective MR tutor; the details of the conditions need further investigation. Tutors can continue learning (Math Knowledge for Teaching and knowledge of MR) through the practice of MR tutoring. This suggests that even when highly skilled tutors may not be available at the beginning of adoption, with experience the tutors can continue to improve in knowledge.

26. Future Research & Broad Implications Future analyses: Investigating MKT relationship to practice and student outcomes. Investigating the relationship between knowledge of MR and practice of MR using fidelity of implementation data. Account for the differences in learning using additional data sources: surveys, teacher lesson plans For future evaluators of teacher “delivered” interventions: Those delivering an intervention can learn from more than just their initial training. This implies that thorough evaluations will have to question static and simplistic views of teachers’ knowledge as something they learn in training and apply in practice.

27. Contact Information: Sarah Green Graduate Student, Dept. of Teaching and Learning Vanderbilt University sarah.green@vanderbilt.edu

28. References: Franke, M. L., Carpenter, T., Fennema, E., Ansell, E., & Behrend, J. (1998). Understanding teachers' self-sustaining, generative change in the context of professional development. Teaching and Teacher Education, 14(1), 67-80. Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers' mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11-30. Remillard, J. T. (2000). Can curriculum materials support teachers' learning? Two fourth-grade teachers' use of a new mathematics text. Elementary School Journal, 100(4), 331-350. Wright, R. J., Martland, J., Stafford, A. K., & Stanger, G. (2006). Teaching number: Advancing children's skills and strategies: Paul Chapman Educational Publishing. Wright, R. J., Martland, J., & Stafford, A. K. (2006). Early numeracy: Assessment for teaching and intervention: Paul Chapman Publishing.