1 / 22

WREC May 29, 2014 Washington, D.C.

Learning “What Works” in Career Pathways Programming: The ISIS Evaluation . David Judkins Abt Associates, Inc. WREC May 29, 2014 Washington, D.C. Abstract.

kristy
Download Presentation

WREC May 29, 2014 Washington, D.C.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning “What Works” in Career Pathways Programming: The ISIS Evaluation . David Judkins Abt Associates, Inc WREC May 29, 2014 Washington, D.C.

  2. Abstract • Introductory review of new methodology for the study of the mediating pathways in the context of randomized experiments of social interventions • Pathways may contain both exposure to program components and early outcomes of that exposure • Other talks in this session talk about randomizing to program components • This talk is about how to analyze when randomization of components is not selected as the evaluation methodology • Plans to apply this new methodology to ISIS

  3. What is ISIS? • Random assignment evaluation of nine “career pathways” programs • Impact, implementation, cost-benefit studies • Evaluation funded by ACF; Abt Associates heading research team • Additional funding for program enhancements/scale up from Open Society Foundations, Joyce Foundation, Kresge Foundation, Meadows Foundation, OFA/ACF (3 HPOG sites) • RA still underway; early impact results in 2-3 years • Programs vary, but all promote access to and completion of post-secondary education, targeting low-income, low-skilled adults and youth • Conducting nine separate studies • Overlap in research questions, measurement, analysis plans

  4. Core Ideas of Career Pathways Programs • Address the wide range of skill and other needs of economically disadvantaged participants • Key “inputs” or elements: Assessment (academic and non-academic); basic and vocational skills training; supports; employment connections • Create manageable, well-articulated training steps • Provide credentials valued in high demand occupations/sectors • Build effective partnerships • Community colleges, employers, CBOs, WIBs

  5. Challenge: Getting inside the “Black Box” • How to determine more effective program components? • Counseling • Tutoring • Financial assistance • Internships • How to determine intermediate outcomes on vital causal paths? • Self-efficacy • Strength of social network • Basic numeracy and literacy skills • Stress management skills

  6. Benefits • Effective components are built into future interventions • Ineffective components are dropped • New components are invented to impact vital early outcomes even more strongly • Future evaluation findings can be obtained more quickly if good early indicators are available

  7. Familiar but Inadequate • Baron and Kenny (1986) mediation triangle • The indirect effect of treatment on Y mediated by M estimated as ab. • The direct effect of treatment on Y is d.

  8. Why Do We Need Better Methodology? • Baron and Kenny assume independence of errors in two equations. • This will be violated if there are any common causes of the two errors such as: • Measured baseline characteristics • Unmeasured baseline characteristics • Other measured mediators • Other unmeasured mediators • Also, Baron and Kenny does not generalize well to categorical outcomes

  9. Pearl’s Method • Judea Pearl has developed a much more powerful and general method. • However, he developed it in a graphical framework unnatural to statisticians in the Neyman-Rubin tradition. • Recently, Kosuke Imai and co-authors have recast it in the language and traditions of Neyman-Rubin. • Dramatically increased popularity • Still very difficult literature to penetrate • Sketch some features today • We plan to use on ISIS

  10. Pearl’s Method Capabilities

  11. ISIS Nomenclature • Blend of nomenclatures proposed by Pearl and by Imai and coworkers. • Key is the concept of forcing an environmental stimulus while “blocking” one type of response to the stimulus • Three potential outcomes per person Y0: person randomly assigned to control Y1: person randomly assigned to treatment and no changes are blocked Y2: person randomly assigned to treatment but change in mediator M is “blocked”

  12. Three Potential Outcomes

  13. Example • Y0 = person’s degree attainment at 36 months if assigned to control • Y1= person’s degree attainment at 36 months if assigned to treatment and no “natural” changes are “blocked” • Y2= person’s degree attainment at 36 months if assigned to treatment but somehow we blocked change to self-confidence

  14. Potential Outcomes Framework for Mediation • Total effect = average of Y1-Y0 • Indirect effect mediated by M = average of Y1-Y2 • Direct effect = average of Y2-Y0 • Then total = indirect + direct • Continuing example: • Y1-Y0 = total effect of treatment on degree attainment • Y1-Y2 = indirect effect of treatment on degree attainment via boosted self-confidence • Y2-Y0 = direct effect of treatment on degree attainment • Note: no need to reference linear models to define the estimands

  15. Mediating Pathways • In ISIS, we are interested in multiple mediating pathways • Trying to decide which are the vital pathways for successor programs to emulate • Multiple mediation can take two forms: • Serial (T causes M1 causes M2 causes Y) • Parallel (T causes both M1 and M2, both of which jointly cause Y) • Pearl’s framework flexible enough to handle simultaneous parallel and serial mediation

  16. Service Exposure and Early Outcomes • Both are mediators • Let D1 and D2 be exposure levels to two program components • Let W1and W2be two early outcomes • Treatment causes D1and D2to change, which causes W1and W2to change (in addition to direct effects of T on them) • All of which lead to changes in Y

  17. Example Graph for Parallel and Serial Mediation

  18. Estimation Process • Build a series of models for every mediator and outcome in the system in terms of causally prior variables • Draw simulated values from system for each pathway of interest • What happens if some services are blocked and some early outcomes are blocked • Pathways with highest simulated Y values are the most promising for future program developers to consider

  19. Assumptions • Strong assumptions required • Considerable debate about best way to express them • Oversimplified version • Variation in M within each level of Y • No unmeasured prime joint causes of M and Y • This includes post-randomization latent outcomes

  20. Example Violation • M = Self-confidence • Y = Degree attainment • Academic skill at unmeasured intermediate point mediates treatment effects on both self-confidence and degree attainment • Easy to come to the mistaken conclusion that shortcut methods work that build self-confidence through means other than skill improvement • Need to measure skill to prevent this incorrect mediational inference

  21. Challenges • Curse of dimensionality • Need to identify graphs to be tested that correspond to program-specific theories of change • Also need to believe that we have measured the necessary set of variables to unconfound the mediator(s) and outcome(s) in the graph of interest

  22. Additional Information David Judkins Principal Scientist Abt Associates David_judkins@abtassoc.com Website www.projectisis.org Brendan Kelly Federal Co-Project Officer, ISIS ACF/OPRE Brendan.Kelly@ACF.hhs.gov Molly Irwin Federal Co-Project Officer, ISIS ACF/OPRE Molly.Irwin@ACF.hhs.gov

More Related