Advanced Study Design

1. Advanced Study Design February 19, 2010

2. Today’s Class • Last Week’s Probing Question • Advanced Study Design • Assignments

3. Probing Question • Let’s say you wanted to do a large-scale research study on boredom • Under what conditions would it be preferable to use • Questionnaire items • Experience sampling method • Quantitative field observations

4. Today’s Class • Last Week’s Probing Question • Advanced Study Design • Assignments

5. Today… • Validity • Validity Threats • Stratification • Counterbalancing and Cross-over Designs • Regression-Discontinuity Designs

6. Validity • Useful jargon...

7. Validity(Trochim & Donnelly, 2007) • Conclusion validity • Internal validity • Construct validity • External validity • Ecological validity

8. Conclusion Validity • The degree to which conclusions you reach about relationships in your data are justified

9. Internal Validity • Assuming that there is a relationship in the study, can you justifiably infer that the relationship is causal?

10. Construct Validity • The degree to which inferences can legitimately be made from the operationalizations in your study, to the theoretical constructs on which those operationalizations were based

11. External Validity • Do your results generalize to other people, procedures, places, and times?

12. Ecological Validity • What is the degree to which the methods, materials, and settings of the study are relevant to natural/legitimate settings?

13. Ecological vs. External Validity • Ecological validity • not about *generalization* to real-life situations • about the whether the "methods, materials and settings" are similar (or identical) to real life. • Ecological validity is about real-world *relevance* • External validity is about generalizability

14. Examples? • High External Validity, Low Ecological Validity • Low External Validity, High Ecological Validity

15. High External Validity, Low Ecological Validity • Lab studies of “seductive details” effect • Instruction that does not include interesting but ultimately irrelevant details leads to better learning, for students of variety of ages performed in lab settings at 2 universities with children of different socio-economic status (SES) & race

16. Low External Validity, High Ecological Validity • A classroom study, with real students, involving legitimate educational tasks, presented in exactly the way a teacher would present them…

17. Low External Validity, High Ecological Validity • A classroom study, with real students, involving legitimate educational tasks, presented in exactly the way a teacher would present them… • With 1 student in each condition

18. Let’s consider a few examples • Vote on which type of validity is violated (any of the five, could be multiple, could even be none) • Explain your reasoning

19. Which type of validity is violated? • Students who read bug messages perform more poorly on post-test • So bug messages hurt learning! You have chosen a categorical variable for the X axis; however, scatterplot graphs can only contain numerical variables.

20. Which type of validity is violated? • I have proven that students learn more Calculus from my Calculus tutoring system • Here is my test, used both pre and post • How well do you know Calculus? 1 2 3 4 5 Not well Very well

21. Which type of validity is violated? • My new tutoring system is much better than the previous tutoring system!

22. Which type of validity is violated? • My new tutoring system is much better than the previous tutoring system!

23. Which type of validity is violated? • I conducted a study comparing my new tutoring system to a previous one • Students who completed the whole tutoring system performed significantly better on post-test in the experimental condition than control condition

24. Which type of validity is violated? • I conducted a study comparing my new tutoring system to a previous one • Students who completed the whole tutoring system performed significantly better on post-test in the experimental condition than control condition • Oops… did I mention only 3% of students completed the whole tutoring system in the control condition?

25. Which type of validity is violated? • Now that I have tested my new learning environment that responds to off-task behavior by giving it to single students in the guidance counselor’s office after school, we can be confident it will work in all school settings

26. Which type of validity is violated? • Now that I have tested my new learning environment with a set of 10 8th graders in Tuktoyaktuk (Northwestern Territory of Canada), all bilingual English-Inuvialuit, with fathers who work in the mine nearby, we can be confident it will work for all students

27. Which type of validity is violated? • Now that I have tested my new learning environment with a set of 120 8th graders in a predominantly middle-class Caucasian suburb of Worcester, we can be confident it will work for all students

28. Some Popular Threats to Internal Validity

29. Maturation Threat • Something happens between pre-test and post-test, aside from your intervention, that impacts student change • E.g. the same thing would have happened whether or not you ran your study

30. Maturation Threat • Something happens between pre-test and post-test, aside from your intervention, that impacts student change • E.g. the same thing would have happened whether or not you ran your study • Any horror stories from your research?

31. Maturation Threat • Something happens between pre-test and post-test, aside from your intervention, that impacts student change • E.g. the same thing would have happened whether or not you ran your study • One teacher taught the same material in class during the same week as the study

32. Mortality Threat • Common in urban classrooms

33. Mortality Threat • Large numbers of participants systematically drop out of the study • Any horror stories from your research?

34. Mortality Threat • Large numbers of participants systematically drop out • Example: I ran a study with homeschool students; response rates were different between conditions

35. Regression to the Mean • If you choose a group based on pre-test performance • The most frequent gamers • The students who scored in the bottom 10% on the pre-test • Some of them were in that group by chance • And can be expected to do better on the post-test

36. Diffusion of Treatment • You assign kids to different conditions, but they see each others’ screens (or talk in the hallway, etc.) • You assign classes randomly to condition within-teacher, but teachers learn strategies from the better condition and use that knowledge in the other condition

37. Diffusion of Treatment • You assign kids to different conditions, but they see each others’ screens (or talk in the hallway, etc.) • You assign classes randomly to condition within-teacher, but teachers learn strategies from the better condition and use that knowledge in the other condition • A major study comparing curricula in Baltimore was called into question because teachers took teaching strategies from the experimental condition to the control condition

38. Compensatory rivalry/resentful demoralization • Students in condition A learn about condition B, which is obviously better • Resentful demoralization – “it’s no fair they got the better software, let’s just quit” • Compensatory rivalry – “we can beat them, even if they got the better software”

39. Compensatory rivalry/resentful demoralization • Students in condition A learn about condition B, which is obviously better • Resentful demoralization – “it’s no fair they got the better software, let’s just quit” • More common for students • Compensatory rivalry – “we can beat them, even if they got the better software” • More common for teachers

40. Confounding • You changed multiple things in your intervention (often inadvertantly), and it’s not clear which change had the impact • Some examples?

41. Confounding • Your meta-cognitive intervention takes longer to go through • Better learning, or just more time-on-task?

42. Comments? Questions?

43. Stratification

44. Pure random sampling • Let’s say you have an intervention that you want to test in 4 groups: urban, wealthy suburban, working-class suburban, and rural students • You have access to students in Worcester, Auburn, Ashburnham, and Cambridge • If you just randomly sample in your population, you are going to get a lot more people from Worcester than Ashburnham • In fact, if you sample 100 people randomly, you have a significant chance of getting nobody at all from Ashburnham

45. Stratification • Your population has N groups • Sample randomly within each group

46. Proportional Stratification • Sample from each group in proportion to its’ size • e.g. randomly select • 5% of all students in Worcester • 5% of all students in Auburn • 5% of all students in Cambridge • 5% of all students in Ashburnham

47. Equalizing Stratification(also called “Disproportionate”) • Sample from each group in proportion to get equal groups • e.g. randomly select • 25 students in Worcester • 25 students in Auburn • 25 students in Cambridge • 25 students in Ashburnham

48. What variables could you stratify on?(in learning sciences)

49. What variables could you stratify on?(in learning sciences) • Gender • Race/Ethnicity • Prior knowledge (pre-test large group, then choose intervention sample) • Disabilities

50. Why? • Why might you want to use • Proportional Stratification • Equalizing Stratification • Good Old Random Sampling