1 / 23

Mica Estrada-Hollenbeck 1 Anna Woodcock 2 David Morella 3 Wesley Schultz 1

Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to Identify a Matched Comparison Group. Mica Estrada-Hollenbeck 1 Anna Woodcock 2 David Morella 3 Wesley Schultz 1 California State University, San Marcos 2 Purdue University

stu
Download Presentation

Mica Estrada-Hollenbeck 1 Anna Woodcock 2 David Morella 3 Wesley Schultz 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to Identify a Matched Comparison Group Mica Estrada-Hollenbeck1 Anna Woodcock2 David Morella3 Wesley Schultz1 California State University, San Marcos 2Purdue University 3Kent State University Presented at the November 14, 2009 AEA Conference, Orlando, Florida.

  2. Research Questions Does participating in the RISE program increase the likelihood that a minority student will pursue a career in the biomedical sciences? Are there some types of students who benefit more from the RISE program than others? Are there elements of the RISE program that are linked with the success of the students? Are the underlying assumptions regarding the efficacy of the elements of the RISE program valid?

  3. The Challenge POPULATION TREATMENT T T P T T P P P P P P P P P P P P P P P P P P

  4. Potential Statistical Solutions When there is no randomized control group • Blocked design • Analysis of covariance • Propensity scores

  5. Propensity Scores Purpose • Corrects for selection bias when randomization is not possible Population Treatment

  6. Propensity Score Definition • Probability that a subject would receive treatment given a set of observed variables. • People with similar propensity scores are similarly likely to receive treatment and can be compared to estimate the effect of treatment. • Rosenbaum and Rubin (1983:420) suggest: • If treatment assignment is strongly ignorable given [covariates used to estimate propensity scores], then the difference between treatment and control means at each value of a [propensity] score is an unbiased estimate of the treatment effect…”

  7. Propensity Score Caveats > Propensity score matching assumes that all variables related to both outcomes and treatment assignment are included in the vector of observed variables (Rosenbaum & Rubin, 1983) > The size of the population/sample from which the propensity scores are derived is important - Minimally, there need to be people in the population who are similar to the treatment group.

  8. Previous Uses • Used primarily with studies retrospectively • Education (Lee and Staff 2007; Wu et al. 2007) • Economics (Dehejia and Wahba 2002, Benjamin 2003, Michalopoulos et al. 2004) • Medical literature (Rubin 1999, Weitzen et al. 2004, Erosheva et al. 2007)

  9. Overview: The Science Study Longitudinal study of minority science students From 45 campuses nationwide, 25 of these have RISE programs 1,380 participants Data collected twice yearly from students Propensity score matched control design Completing fourth year (8 waves of data)

  10. Longitudinal Panel (at recruitment)

  11. Longitudinal Panel • 72% Female • Ethnicity/Race: • 49% African American • 39% Hispanic/Latino(a) • 1% Native American • Major: • 63% Biological Sciences • 21% Natural Sciences • 12% Behavioral & Social Sciences • 4% Mathematics & Engineering

  12. Survey Data Collection Data collected through secure web interface www.TheScienceStudy.com

  13. Propensity Score Generation • Identify what variables are predictive of treatment group • -- Must be measurable items from both population and treatment sample • -- In a prospective longitudinal study, we use the first data collected to identify the groups. • 2. Using variables as predictors, a propensity score is generated for each person in the sample (logistic regression, SPSS). • 3. Recruited match pool using faculty referrals: • - RISE funded: N=750 possible, recruited 402 • (added new students in W2 and W4) • - Match pool: N=2166, wanted 402 (plus overmatch)

  14. Propensity Score Generation • Additionally, age squared and interactions of Gender and GPA, Gender and Transfer Student Status, and Gender by educational status were added.

  15. Propensity Scores • Everyone received a score Population Treatment =.94 =.85 =.75 =.93 =.67 =.93 =.97 =.87 =.85 =.93 =.93 =.75 =.85 =.75 =.85 =.93 =.93 =.85 =.93 =.67 =.67 =.67 =.93 =.75 =.75 =.93

  16. Propensity Scores Matching Process

  17. Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded (N=101), and MATCH = students never funded (N=200) by any program and enrolled on a RISE campus. Dropped = students who were at one time enrolled in RISE but did not complete it. Intention to pursue career as biomedical scientist.

  18. Baccalaureate Graduation (Fall, 08) • No difference between Dropped & Match, χ2(1)=0.58, p=.81 • Significant difference between RISE and Combined Dropped/Match, χ2(1)=10.37, p=.001

  19. Graduate School: Applications & Offers Applications: Science-related Programs Offers: Science-related Programs

  20. Post Baccalaureate Outcomes (Fall, 08) Enrollment Attainment • 21 panel members with a Ph.D. (W6, 12/08) • 1 panel member MD • 2010/2011 first large wave of Ph.D. eligible panel members

  21. Thank You

  22. Bibliography Benjamin, Daniel. 2003. “Does 401(k) Eligibility Increase Savings? Evidence from Propensity Score Classification.” Journal of Public Economics 87: 1259-1290. Dehejia, Rajeev and Sadek Wahba. 2002. “Propensity Score Matching Methods for Non-Experimental Causal Studies.” Review of Economics and Statistics 84 (1): 151-161. Lee, Jennifer C. and Jeremy Staff. 2007. “When Work Matters: The Varying Impact of Work Intensity on High School Dropout.” Sociology of Education 80 (2): 158-178. Michalopoulos, Charles, Howard S. Bloom ­ Carolyn J. Hill. 2004. “Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?” Review of Economics and Statistics 86 (1): 156-179. Rosenbaum, Paul and Donald Rubin. 1983. “The Central Role of the Propensity Score in Observation Studies for Causal Effects.” Biometrika 70 (1): 41-56. Rubin, Donald. 2001. “Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation.” Health Services and Outcomes Research Methodology 2: 169-188. Weitzen, Sherry, Kate Lapane, Alicia Toledano, Anne Hume and Vincent Mor. 2004. “Principles for Modeling Propensity Scores in Medical Research: A Systemic Literature Review.” Pharmacoepidemiolgy and Drug Safety 13: 841-853. Wu, Wei, Stephen West, and Jan Hughes. 2007. “Short-Term Effects of Grade Retention of the Growth Rate of Woodcock Johnson III Broad Math and Reading Scores.” Journal of School Psychology.

More Related