1 / 54

Presentation Outline

Addressing the Don’t Ask, Don’t Tell Practice in Observational Studies: Using Interviews to Understand the Assignment Mechanism Jordan H. Rickles Social Research Methodology Division Graduate School of Education & Information Studies University of California, Los Angeles

draco
Download Presentation

Presentation Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Addressing the Don’t Ask, Don’t Tell Practice in Observational Studies: Using Interviews to Understand the Assignment Mechanism Jordan H. Rickles Social Research Methodology Division Graduate School of Education & Information Studies University of California, Los Angeles SREE Annual Meeting | Washington, DC | March 6, 2010

  2. Presentation Outline • Overview of Study • Brief Overview of Causal Inference • The Assignment Mechanism • Literature on Course Assignment • Data and Methods • Interview Findings • Example at Three Schools • Implications for Estimating Effects • Conclusions Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 2

  3. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions How many legs does a dog have if you call the tail a leg? Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 3

  4. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Two Objectives • Draw attention to the importance of studying the assignment mechanism • Statistical inference for causal effects “requires the specification of a posited assignment mechanism describing the process by which treatments were assigned to units” (Rubin, 1991, p. 403). Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 4

  5. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Two Objectives • Draw attention to the importance of studying the assignment mechanism • Statistical inference for causal effects “requires the specification of a posited assignment mechanism describing the process by which treatments were assigned to units” (Rubin, 1991, p. 403). • “Knowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measured” (Cook, Shadish & Wong, 2008, p. 740). Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 5

  6. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Two Objectives • Draw attention to the importance of studying the assignment mechanism • Statistical inference for causal effects “requires the specification of a posited assignment mechanism describing the process by which treatments were assigned to units” (Rubin, 1991, p. 403). • “Knowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measured” (Cook, Shadish & Wong, 2008, p. 740). • In randomized controlled trial and regression discontinuity designs assignment mechanism is usually known Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 6

  7. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Two Objectives • Draw attention to the importance of studying the assignment mechanism • Statistical inference for causal effects “requires the specification of a posited assignment mechanism describing the process by which treatments were assigned to units” (Rubin, 1991, p. 403). • “Knowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measured” (Cook, Shadish & Wong, 2008, p. 740). • In randomized controlled trial and regression discontinuity designs assignment mechanism is usually known • In observational study the assignment mechanism is usually unknown Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 7

  8. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Two Objectives • Provide example for using interviews to study the assignment mechanism • Focus on study of taking algebra vs. pre-algebra in 8th grade Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 8

  9. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The Potential Outcomes Framework • Causal effects defined by potential outcomes at the unit of analysis • δi= yt(i) – yc(i) • Average Treatment Effect: E[δATE] = E[Yt] – E[Yc] Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 9

  10. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The Potential Outcomes Framework • Causal effects defined by potential outcomes at the unit of analysis • δi= yt(i) – yc(i) • Average Treatment Effect: E[δATE] = E[Yt] – E[Yc] • Fundamental problem of causal inference (Holland, 1986) • Can only observe yt for units assigned to the treatment and can only observe yc for units assigned to the control • Under selection independence can estimate ATE as • E[δATE]=E[Yt|D=1] – E[Yc|D=0] Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 10

  11. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods | Interview Findings | Example at Three Schools | Implications | Conclusions • Selection Independence • Treatment assignment is independent of the potential outcomes • Holds for a randomized experiment • Not likely to hold for an observational study where treatment assignment depends on factors (S) that are associated with potential outcomes Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 11

  12. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods | Interview Findings | Example at Three Schools | Implications | Conclusions • Selection Independence • Treatment assignment is independent of the potential outcomes • Holds for a randomized experiment • Not likely to hold for an observational study where treatment assignment depends on factors (S) that are associated with potential outcomes • Assumption of strongly ignorable treatment assignment (Rosenbaum & Rubin, 1983) • Treatment assignment is independent of the potential outcomes conditional on observed factors (S) • Under conditional independence can estimate ATE as • E[δATE]=E[Yt|S, D=1] – E[Yc|S, D=0] Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 12

  13. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods | Interview Findings | Example at Three Schools | Implications | Conclusions • Validity of the assumption of strongly ignorable treatment assignment depends on the assignment mechanism • If methods for conditioning on S accurately capture the true assignment mechanism then assumption holds Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 13

  14. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods | Interview Findings | Example at Three Schools | Implications | Conclusions • Validity of the assumption of strongly ignorable treatment assignment depends on the assignment mechanism • If methods for conditioning on S accurately capture the true assignment mechanism then assumption holds • If methods for conditioning on S do not fully capture the true assignment mechanism then assumption may not hold Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 14

  15. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods | Interview Findings | Example at Three Schools | Implications | Conclusions • Validity of the assumption of strongly ignorable treatment assignment depends on the assignment mechanism • If methods for conditioning on S accurately capture the true assignment mechanism then assumption holds • If methods for conditioning on S do not fully capture the true assignment mechanism then assumption may not hold • How do we know if assumption holds? • Short answer: we don’t. • Better answer: we can investigate the assignment mechanism for a more informed determination Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 15

  16. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • What does the literature say about the assignment of 8th graders to algebra vs. pre-algebra? • Need to rely on related literature on ability grouping, tracking, and high school course taking Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 16

  17. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • What does the literature say about the assignment of 8th graders to algebra vs. pre-algebra? • Need to rely on related literature on ability grouping, tracking, and high school course taking • Rational choice (or human capital) theory: students are matched with courses to efficiently accommodate differences in student ability Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 17

  18. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • What does the literature say about the assignment of 8th graders to algebra vs. pre-algebra? • Need to rely on related literature on ability grouping, tracking, and high school course taking • Rational choice (or human capital) theory: students are matched with courses to efficiently accommodate differences in student ability • Nonacademic factors can mediate or constrain the optimality of the rational choice theory • e.g., social class, student expectations, teacher & parent input, school assignment practices • See Hallinan (1994) and Oakes, Gamoran & Page (1992) Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 18

  19. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • How have past researchers estimated the effect of 8th grade algebra? • Mostly through regression-based modeling adjustments based on a variety of covariates • OLS regression (Gamoran & Hannigan, 2000) • Path analysis (Smith, 1996) • Linear growth modeling (Ma, 2005) Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 19

  20. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • How have past researchers estimated the effect of 8th grade algebra? • Mostly through regression-based modeling adjustments based on a variety of covariates • OLS regression (Gamoran & Hannigan, 2000) • Path analysis (Smith, 1996) • Linear growth modeling (Ma, 2005) • Studies generally find positive effect of 8th grade algebra on subsequent mathematics achievement • Findings depend on assumption of strong ignorability • Studies do not examine plausibility of this assumption Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 20

  21. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Data for estimation of causal effects • Student-level data for a cohort of 2006-07 8th graders • From large urban California school district • Academic & demographic data from district’s administrative files, covering 2002-03 through 2007-08 school years Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 21

  22. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Data for estimation of causal effects • Student-level data for a cohort of 2006-07 8th graders • From large urban California school district • Academic & demographic data from district’s administrative files, covering 2002-03 through 2007-08 school years • Data to examine the assignment process • 10 interviews of key school-level decision makers • From middle schools with 8th graders in both algebra and pre-algebra courses • Interviews conducted in early 2009 Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 22

  23. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The interview protocol • Part I: six semi-structured, open-ended questions about: • The decision-making process • The types of students typically assigned to algebra • The general philosophy regarding 8th grade algebra Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 23

  24. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The interview protocol • Part I: six semi-structured, open-ended questions about: • The decision-making process • The types of students typically assigned to algebra • The general philosophy regarding 8th grade algebra • Part II: questions about two student scenarios • Designed to get more standardized summary of assignment mechanism across schools Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 24

  25. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The interview protocol • Part I: six semi-structured, open-ended questions about: • The decision-making process • The types of students typically assigned to algebra • The general philosophy regarding 8th grade algebra • Part II: questions about two student scenarios • Designed to get more standardized summary of assignment mechanism across schools • Part III: rate importance of different information sources in assignment decision • 5 category Likert scale Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 25

  26. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The interview protocol • Part II, Scenario 1: Martin • In 7th grade, Martin got a C in his first semester math class and a D the second semester. He received C’s and B’s in his other classes. He also received a mix of satisfactory and unsatisfactory marks for work habits and cooperation. In 6th grade, Martin’s math grades were a little higher, with a C the first semester and a B the second semester. Similarly, he scored Basic on the 6th grade math CST and Below Basic on the 7th grade math CST. You heard a couple of Martin’s 7th grade teachers mention that he started slipping behind and became more of a disruption in class as the year progressed. • On a scale of 1 to 5, how likely is it that you would enroll Martin in Algebra in 8th grade? Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 26

  27. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • The interview protocol • Part II, Scenario 2: Maya • Maya moved to California from Mexico during her 6th grade year and started attending this school in 7th grade. She is an English learner and is struggling to keep up in most of her classes. She received mostly D’s in 7th grade, but got a C in her second semester math class. Her work habits and cooperation marks are all satisfactory and there is no mention of any disciplinary problems in her records. She scored Far Below Basic on her 6th grade math and ELA CST tests but scored Basic on her 7th grade math test. You do not know much else about her except what is in her official record. • On a scale of 1 to 5, how likely is it that you would enroll Maya in Algebra in 8th grade? Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 27

  28. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Analysis of interviews • Coded each interview based on four assignment process characteristics • Whether weight is given to objective (e.g., test scores) or subjective (e.g., teacher recommendations) criteria • Whether decisions are primarily based on data systematically collected by the district (i.e., observable) or non-systematic data (i.e., unobservable). • Whether well defined inclusion/exclusion decision rules are used or not • Whether the school’s course placement philosophy is more protectionist or laissez faire Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 28

  29. Overview | Causal Inference | Assignment Mechanism | Literature | Data & Methods Interview Findings | Example at Three Schools | Implications | Conclusions • Analysis of interviews • Coded each interview based on four assignment process characteristics • Whether weight is given to objective (e.g., test scores) or subjective (e.g., teacher recommendations) criteria • Whether decisions are primarily based on data systematically collected by the district (i.e., observable) or non-systematic data (i.e., unobservable). • Whether well defined inclusion/exclusion decision rules are used or not • Whether the school’s course placement philosophy is more protectionist or laissez faire • Combined the codes with the close-ended responses for analysis Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 29

  30. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Schools draw from a common battery of information sources in the decision-making process Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 30

  31. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Schools differ in weight given to specific information sources and the benchmarks (or cut-points) used to determine algebra placement • Differences exemplified in responses to scenarios Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 31

  32. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Underlying philosophy associated with strictness of algebra placement benchmarks and decisions for “borderline” students • Three schools had a protectionist philosophy • Do not want to “program a kid for failure” • “We don’t want students to be in a class where they’re going to struggle so much that they’re not going to be successful.” Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 32

  33. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Underlying philosophy associated with strictness of algebra placement benchmarks and decisions for “borderline” students • Three schools had a protectionist philosophy • Do not want to “program a kid for failure” • “We don’t want students to be in a class where they’re going to struggle so much that they’re not going to be successful.” • Five schools had a laissez faire philosophy • “Kids have a right to fail.” • A student is “still better off failing the on-grade-level class than if he’d taken the other one.” Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 33

  34. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Can exemplify school differences in the assignment mechanism by contrasting practices at three schools • Haverbrook Middle School (HMS) • Half of the 600 8th graders in algebra • Teacher recommendations, math course grades & MDTP most important sources of information • Soft benchmark of 70% on MDTP, C in 7th grade math class & Basic on CST • Not strongly protectionist or laissez faire • Martin in algebra, need more info to place Maya Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 34

  35. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Can exemplify school differences in the assignment mechanism by contrasting practices at three schools • Ogden Middle School (OMS) • Half of the 900 8th graders in algebra • Teacher recommendations, CST & weekly school-developed math quizzes most important sources of information • No formal benchmarks but look for 60% correct on quizzes • laissez faire philosophy • Martin in pre-algebra, need more info to place Maya Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 35

  36. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Can exemplify school differences in the assignment mechanism by contrasting practices at three schools • Shelby Middle School (SMS) • One-third of the 500 8th graders in algebra • CST, math grades & MDTP most important sources of information • Strong benchmarks of 80% on MDTP, Advanced on CST, B in 7th grade math class • Protectionist philosophy • Martin and Maya in pre-algebra Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 36

  37. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Distributions of prior achievement for algebra and pre-algebra students at the three schools are consistent with the general selection process described by each school Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 37

  38. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Observed probability of algebra placement given CST performance and course grades also reflects school’s reported selection process • Shelby Middle School (SMS) Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 38

  39. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Observed probability of algebra placement given CST performance and course grades also reflects school’s reported selection process • Ogden Middle School (OMS) Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 39

  40. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Observed probability of algebra placement given CST performance and course grades also reflects school’s reported selection process • HaverbrookMiddle School (HMS) Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 40

  41. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Investigating the assignment mechanism through interviews aides the causal effect research design • Provides current & localized information about which factors are associated with treatment assignment • What confounders should be controlled for? Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 41

  42. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Investigating the assignment mechanism through interviews aides the causal effect research design • Provides current & localized information about which factors are associated with treatment assignment • What confounders should be controlled for? • Provides information about how the factors are associated with treatment assignment • How should confounders get included in a statistical model? Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 42

  43. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Investigating the assignment mechanism through interviews aides the causal effect research design • Provides current & localized information about which factors are associated with treatment assignment • What confounders should be controlled for? • Provides information about how the factors are associated with treatment assignment • How should confounders get included in a statistical model? • Provides information about heterogeneity in the assignment process across schools • Can a single fixed-effects model control for confounders at all sites? Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 43

  44. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Example: specifying a logistic regression model for assignment to algebra vs. pre-algebra • Four different model specifications • Model 1: naïve model with school homogeneity • Model 2: informed model with school homogeneity • Model 3: informed model with random intercept • Model 4: informed model with random intercept & slopes • Want to use predicted probabilities for propensity score-based causal effect estimates, but need to use most appropriate propensity score model Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 44

  45. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Different models result in different propensity scores • For the average student … Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 45

  46. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Different models result in different propensity scores • For a student like Martin … Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 46

  47. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Different models result in different propensity scores • For a student like Maya … Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 47

  48. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • Bias resulting for unobserved key factors should be tested through sensitivity analysis • MDTP not observed • Probably correlated with observed CST scores • Teacher recommendations not observed • Probably correlated with observed course grades Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 48

  49. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • When the assignment mechanism is unknown, investigating the assignment process can help control for selection bias and communicate potential sources of bias Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 49

  50. Overview | Causal Inference | Assignment Mechanism | Literature | Data & MethodsInterview Findings | Example at Three Schools | Implications | Conclusions • When the assignment mechanism is unknown, investigating the assignment process can help control for selection bias and communicate potential sources of bias • An investigation of the assignment mechanism does not have to be extensive or resource intensive • Short interviews conducted in a small sample of schools provides a lot of information • Interviews could be part of a pilot study Using Interviews to Understand the Assignment Mechanism | Rickles | SREE 2010 | Slide 50

More Related