1 / 11

The Value of Random Assignment Impact Evaluations for Youth-Serving Interventions?

This article discusses the value of random assignment impact evaluations for youth-serving interventions, using career academy research as a case study. It highlights the importance of conducting impact evaluations, the benefits of randomized assignment, and the conditions necessary for a fair test. The findings from the career academy evaluation are also presented, demonstrating the potential of career-related experiences in improving post-secondary labor market prospects.

ewalters
Download Presentation

The Value of Random Assignment Impact Evaluations for Youth-Serving Interventions?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Value of Random Assignment Impact Evaluations for Youth-Serving Interventions? Notes from Career Academy Research and Practice James Kemple Senior Fellow, MDRC www.mdrc.org

  2. What Can Career Academy Research and Practice Offer Evidence-Based Policy? Practice • 34-year track record of implementation, planned expansion, and efforts at continuous improvement • Intervention with goals and core features aligned with important problems in high schools and prominent policy options Research • 25 years of non-experimental research and a commitment to learning what works • 10-year random assignment field experiment involving 9 sites, over 1,700 students, and 8-years of follow-up • Positive effects on labor market outcomes without compromising on academic goals

  3. Context for Impact Evaluations • Learning “what works” is a long-term and cumulative process • Questions drive methodology, not the reverse • Multiple questions require multiple methods • Must balance research ambition against operational and political realities • Knowledge-building should be an integral part of policy development and continuous improvement, not an add-on or after-thought

  4. Why Conduct Impact Evaluations? • Outcomes vs. Impacts • Outcome: Measure of individual or group behavior, attitudes, achievement, labor market participation, an so on. • Impact: The effect on an intervention on an outcome: the difference between outcome for program group and outcome for counterfactual. • Outcome-focused studies risk getting the wrong answer to the right question • Outcome standards risk awarding programs: • based on who they serve, rather than what they do • that operate under promising conditions, rather than use promising practices

  5. National Averages for Similar Students in Similar Schools 100 Evaluation Sample 90 8.6 80 70 60 Percent Graduating On-Time 50 80.4 40 72.9 72.2 63.3 30 48.6 20 10 0 Academy Non-Academy Career/Tech. General Academic Academy Judging Program Impacts:High Outcomes/No Impact Note: National average estimates are adjusted to represent a sample with the same background characteristics as those in the Evaluation Sample.

  6. 100 100 100 90 90 90 85 85 84 ) ) 80 80 80 ) 72 71 70 70 70 60 60 60 53 50 50 50 Graduation Rate (% Graduation Rates (% Graduation Rates (% 40 40 40 30 30 30 20 20 20 10 10 10 0 0 0 Program A Program B Program C Academy Group Control Group Judging Program Impacts:Adding value vs. starting with strong context

  7. Career Academy Impacts on Average Earnings Impacts

  8. Guiding Principles for Impact Evaluations • Random assignment may be the “gold standard” but it is not the “Philosopher’s Stone” (i.e., won’t extend life or answer every important question.) • Questions drive methodology, not the reverse • Because evaluations involve multiple questions they require multiple methods • Implementing methodologies requires balancing research ambition against operational realities • Strong research designs cannot compensate for weak treatments

  9. Conditions for Random Assignment • Priority Question: What is the impact? • Ethical and legal standards • No denial of services to which otherwise entitled • No reduction in expected service levels • Informed consent and data confidentiality • Operational Realities • Collaboration between researchers and program managers • Structured process for program entry or access to resources • Excess demand: more eligible applicants than available program slots or resources • Fair method for allocating scarce resources • Opportunity for a “fair test” of the intervention

  10. Conditions for a “Fair Test” • Strong contrast with “status quo” • Implementation of program being tested • Participant exposure to program services • Well-understood alternative to program service • High quality methods for answering questions about why programs are effective (or not) and for whom • Dissemination of findings about what works and what does not work

  11. Implications of Career Academies Evaluation • Random assignment provided findings that could not have been obtained with other designs. • Increased investments in career-related experiences during high school can improve post-secondary labor market prospects. • Career Academies serve as viable pathway to post-secondary education, but not necessarily better than other opportunities. • Career Academies demonstrate feasibility of accomplishing goals of school-to-career and career technical education without compromising on academic goals.

More Related