Conducting Rigorous Evaluations of Interventions That (May) Improve Student Learning: A Researcher/Teacher Reflects

1. Conducting Rigorous Evaluations of Interventions That (May) Improve Student Learning:A Researcher/Teacher Reflects Jon R. Star Michigan State University Harvard University (as of July 2007)

2. About me • Former middle and high school mathematics teacher • PhD in Educational Psychology, with emphasis on middle/high school students’ learning of mathematics • Two grants from US Dept. of Ed (IES) exploring interventions designed to improve students’ learning of mathematics

3. Why I was asked to speak today • Perhaps useful for me to share my experiences as a researcher who designs and conducts rigorous (experimental) evaluations of interventions designed to improve students’ learning of mathematics • What are the challenges that I face in my work that I feel are endemic to the evaluation work that many of you are trying to do?

4. Example 1 Example 1 Example 2 p.1 p.1 Example 2 p.2 Research examples (1) • The benefits of comparison for learning mathematics • Better to solve problems by viewing worked examples sequentially or side by side?

5. Then If ? ? ? = Proportion Problem Schemata ? ? ? ? Ratio Problem Schemata Research examples (2) • Learning strategies for solving ratio and proportion problems using schema • Is using schema is better than not using schema?

6. Two types of challenges • Tension between: • my role as a researcher, and • my heart and soul as a teacher • Balancing between: • the need to design the intervention to be very ‘clean’ to enable an experimental study, and • the need to provide experiences for students that are appreciative of good (and sometimes ‘messy’) instructional practices

7. Researcher vs.teacher

8. Flow of intervention • Let intervention run its course • Give help in standardized way • Don’t deviate from the instructional protocol or script • If students don’t ‘get it’, spend extra time with them • Focus on mastery of the material • Spend extra time with struggling students

9. = Scorecard Flow of intervention

10. Assessing learning • Materials need to discriminate and have room for learning • OK with pretest scores of 10% • OK with posttest scores of 50% • Materials should be challenging but students should be able to succeed • ‘Failing’ tests is disheartening • Average posttest grade of a B?

11. = Flow of intervention Scorecard Assessing learning

12. Relations with schools • Keep professional distance from schools and teachers • Don’t want to ‘spill the beans’ so that we can use the school in future work • Don’t tell if study didn’t work • Establish deep and long-lasting relationships with schools, teachers, and students • Researchers shouldn’t zip in/out • Go back and visit; share results

13. = Flow of intervention Assessing learning Scorecard Relations with schools

14. Ethics of study design • Experimental design tells us the most about the intervention • Control group is necessary • There may be “winner” and “loser” conditions • All conditions should lead to student learning • Design should allow non-treatment students to experience treatment eventually (if successful)

15. = Flow of intervention Assessing learning Relations with schools Scorecard Ethics of study design

16. Two types of challenges • Tension between: • my role as a researcher, and • my heart and soul as a teacher • Balancing between: • the need to design the intervention to be very ‘clean’ to enable an experimental study, and • the need to provide experiences for students that are appreciative of good (and sometimes ‘messy’) instructional practices

17. Clean vs.messy

18. Instructional format • Students working alone is an easier design • Logistically • Statistically • Students working with partners is often better for learning • Also can help teachers get more comfortable with this instructional format

19. = Scorecard Instructional format

20. Instructional delivery • Researchers providing instruction is easier: • no PD • consistency across schools • Letting students’ regular teachers do the teaching is better: • we are unfamiliar with norms • we don’t know students’ names

21. = Instructional format Scorecard Instructional delivery

22. Relations with teachers • Better if teachers don’t know too much about the intervention • might bias their interactions with students • The best way to have a long term impact on students is by working with teachers on instructional improvement

23. = Instructional format Instructional delivery Scorecard Relations with teachers

24. Thanks! Jon Star jonstar@msu.edu www.msu.edu/~jonstar