1 / 35

A Framework for Studying Variation in Program Effects

Explore the factors influencing program effectiveness, such as treatment contrast, client characteristics, and program context. Understand how these variables impact outcomes and study sources of variation.

lopp
Download Presentation

A Framework for Studying Variation in Program Effects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Framework for StudyingVariation in Program Effects June 14, 2015 Michael J. Weiss Based on: Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A Conceptual Framework for Studying the Sources of Variation in Program Effects. Journal of Policy Analysis and Management, 33(3).

  2. Outline Section I: Introduction Section II: Proximal Sources of Variation in Program Effects Section III: Connecting the Treatment Contrast to Program Implementation & Treatment Fidelity

  3. Section 1 Introduction

  4. Consequences of Effect Variation • Targeting services (types of clients / locations) • Equity / fairness • Best case / worst case scenarios • Generalizability / Scaling • Research design • What research questions to address? • Sample size • Etc.

  5. Three Motivating Studies • After School Programs • certain types “work” (sequenced activities, active learning, limited # of goals, activities tied to goals) • Charter Schools • certain types “work” (clear behavior standards, zero-tolerance, coaching new teachers) • Welfare-To-Work • certain types “work” (counseling to get jobs quick, personalized attention) • context mattered (low unemployment rate  larger effects)

  6. Section 1: Introduction DEFINING A PROGRAM EFFECT for an individual

  7. Potential Outcome Weight Loss Program Neyman Program Effect On Neyman No Program Potential Outcome

  8. Section 1: Introduction Estimating the average PROGRAM EFFECT for a Group of individuals

  9. Estimating the AverageProgram Effect Randomize individuals (or groups) to: • Receive program offer • Not receive program offer Compare two group’s average future outcomes

  10. Section 1: Introduction Defining Variation in Program Effects

  11. Potential Outcome Weight Loss Program Rubin Neyman Program Effect On Neyman Program Effect On Rubin No Program Potential Outcome Potential Outcome Weight Loss Program No Program Potential Outcome

  12. The Framework Treatment Fidelity Take-up Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Program Effect Treatment Contrast Treatment Received Outcomes Treatment Offered Take-up Characteristics of Implementing Organization Client Characteristicsand Context

  13. Section 2 Proximal Sources of Variation in Program Effects

  14. The Three C’s • Treatment Contrast • Client Characteristics • Program Context

  15. Section 2: Proximal Sources of Variation in Program Effects 1. The Treatment Contrast

  16. Potential Outcome Weight Loss Program Neyman Neyman Program Effect On Neyman Contrast 2 Program Effect On Neyman Contrast 1 No Program Potential Outcome Potential Outcome Weight Loss Program Alternate Program Potential Outcome

  17. Treatment Contrast and Program Effect Treatment Fidelity Treatment Received Outcomes Take-up Treatment Planned Outcomes Outcomes Treatment Offered Treatment Received Treatment Received Mediators - Employment - Achievement - Recidivism - etc. - Employment - Achievement - Recidivism - etc. Program Model - Content - Quantity - Quality - Conveyance - Content - Quantity - Quality - Conveyance Implementation Plan Implementation Process Program Effect Treatment Contrast Treatment Fidelity Treatment Received Outcomes Mediators Treatment Planned Treatment Offered Program Model Implementation Plan Implementation Process Take-up Characteristics of Implementing Organization Client Characteristicsand Context

  18. Section 2: Proximal Sources of Variation in Program Effects 2. Client Characteristics

  19. Potential Outcome Weight Loss Program Low-Risk High-Risk Program Effect On Low-risk Type Program Effect On High-risk Type No Program Potential Outcome Potential Outcome Weight Loss Program No Program Potential Outcome

  20. Client Characteristics as Moderators Treatment Fidelity Take-up Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Program Effect Treatment Contrast Treatment Fidelity Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Take-up Characteristics of Implementing Organization Client Characteristics and Context

  21. Author’s Suggestions (Client Characteristics as Moderators) • Risk • E.g., pre-test, pre-RA income level • Readiness • E.g., acknowledgment of problem (substance abuse) • Program Theory • E.g., age (adult education that relies on technology)

  22. Section 2: Proximal Sources of Variation in Program Effects 3. Program Context

  23. Potential Outcome Weight Loss Program Neyman Neyman Program Effect On Neyman Context 1 Program Effect On Neyman Context 2 No Program Potential Outcome Potential Outcome Weight Loss Program No Program Potential Outcome

  24. Context as a Moderator Treatment Fidelity Take-up Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Program Effect Treatment Contrast Treatment Fidelity Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Take-up Characteristics of Implementing Organization Client Characteristics and Context

  25. Section 3 Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity

  26. Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity From Treatment Offered to Treatment Received: Client Take-up

  27. Client Take-Up Treatment Fidelity Take-up Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Program Effect Treatment Contrast Treatment Fidelity Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Take-up Characteristics of Implementing Organization Client Characteristics and Context

  28. Author’s Suggestions (Strategies to Increase Client Take-Up) • Incentives / Inducements • ASAP’s MetroCard tied to attending advising sessions • Requirements / Mandates • Opening Doors required student success course • Remove Barriers • H&R Block’s FAFSA simplification • 401(k) saving plans “opt-out” • Improved Outreach • High school to college transition, summer outreach

  29. Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity From Treatment Planned to Treatment Offered: Program Implementation

  30. Program Implementation Treatment Fidelity 1 Take-up Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process 2 Program Effect Treatment Contrast Treatment Fidelity Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Take-up 3 Characteristics of Implementing Organization Client Characteristics and Context

  31. Treatment Plan 1 Definition: Planned content, quantity, quality & conveyance of treatment services for clients • The plans for “what, how much, how, and by whom?” of treatment • Varies in specificity

  32. Implementation Plan 2 Definition: Set of instructions for how the treatment plan is to be realized Worth Considering: • Staff recruitment and selection • Staff training • Staff monitoring and supports

  33. Implementing Organization(Author’s Suggestions ) 3 • Strong Leadership • Resources / Capacity • Supportive Culture / Climate • Involvement of Outside Monitor or “Fixer”

  34. Treatment Fidelity Treatment Fidelity Take-up Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Program Effect Treatment Contrast Treatment Fidelity Treatment Planned Treatment Received Outcomes Treatment Offered Program Model Implementation Plan Implementation Process Take-up Characteristics of Implementing Organization Client Characteristics and Context

  35. A Framework for StudyingVariation in Program Effects Michael J. Weiss Howard S. Bloom Thomas Brock michael.weiss@mdrc.orghoward.bloom@mdrc.orgthomas.brock@ed.gov

More Related