1 / 25

My Role in ERA

Designing a Random Assignment Social Experiment In the U.K.; The Employment Retention and Advancement Demonstration (ERA). My Role in ERA. A member of the 6-person team in the British Cabinet Office that: Designed the ERA program Designed the evaluation of ERA to determine how effective it is

elias
Download Presentation

My Role in ERA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Designing a Random Assignment Social Experiment In the U.K.; The Employment Retention and Advancement Demonstration (ERA)

  2. My Role in ERA • A member of the 6-person team in the British Cabinet Office that: • Designed the ERA program • Designed the evaluation of ERA to determine how effective it is • A member of the team evaluating ERA • I’m in charge of the cost-benefit analysis

  3. My Talk Today • How it was designed • What the ERA program is • The ERA evaluation and how experimental methods are being used to evaluated it • Some early results • Some lessons from designing ERA (if time permits)

  4. Employment Retention and Advancement Demonstration (ERA) • Design Work in 2001-2002 • Run as pilot program in 2003-2007 in 6 sites • Analysis conducted: 2007-2010

  5. Why Was ERA Undertaken? • To test a program that tries to keep low-wage workers employed after they find jobs and help them advance • To promote the use of random assignment experiments in the U.K. • Because it was expected the value of the information obtained about the program would exceed the cost of the obtaining the information (“the rational paradigm) • This requires that the information is actually used

  6. Program Features • Continued contact with Jobcentre Plus advisors (ASAs) after obtaining employment • Retention bonus of £400 every 17 weeks if works full-time for 13 weeks • Training bonus of £8 per hour of training • Must contact ASA to receive bonus • ASAs advise participants on job advancement

  7. Unique Features of ERA • Design of Program and Evaluation done simultaneously • Design work done at British Cabinet Office • Expectation that decision on national implementation of pilot would be based on evaluation findings • Pilot run as large-scale random assignment experiment • Focus is on what happens after a job is obtained

  8. Evaluation Components • Process or implementation study • Impact analysis • Cost study • Cost-benefit analysis

  9. Project Characteristics • Long planning period • Absence of political pressure • Developing the program and evaluation designs in tandem • Allowed for a good evaluation design • Designing the project in the U.K. Cabinet Office • Permitted project team to focus on the project • Need to transfer project to DWP once designed • Random assignment

  10. How Does Random Assignment Work? • Estimate the impact of a policy, a change in a program, or a intervention • Provide evidence of whether the policy has led to (or caused) the change it was designed to - a causal link! • The overall objective - to provide policymakers with evidence of whether their policy works

  11. Measuring Impact • Experiments measure the impact of policies/interventions in terms of their impact on outcomes; • E.g. Does ERA increase earnings? • The outcome measure is the earnings of the program group

  12. Establishing Causality • To establish that a policy or intervention has caused change to occur rather than some other factor • E.g. many factors will affect individuals earnings in addition to participation in ERA • If we find that earnings have increased among the program group, how do we rule-out the influence of other factors? • E.g. most persons who entered ERA were not employed, but some would inevitably have found jobs without ERA. • We do this by estimating what we call the ‘counterfactual’—what would have happened without the program

  13. The Counterfactual • Is what would have occurred in the absence of the policy or intervention • E.g. what would have happened to earnings over the same period of time for the same individuals had ERA not existed? • This is unobservable or missing information • We have to estimate the counterfactual - that is determine what would have happened in the absence of ERA

  14. Estimating the Counterfactual • Wide range of ways to do this • These vary in complexity, rigor and the degree of control required by those planning the evaluation • We will look at only one method: The simple two group random assignment experiment • When feasible, random assignment is the best method for estimating a counterfactual

  15. Random Assignment • In theory the most rigorous way to assess the impact of a policy or intervention • Provides unbiased estimates of intervention impacts • How does it work? • You identify individuals or groups who are eligible for a new intervention or policy • You create two groups at random - intervention and control groups (essentially a computer flips a coin) • Intervention group receives the new service or intervention, control group does not.

  16. Simple Randomised Experiment Intervention R Eligible population Intervention group Outcome = O1 Control group Outcome = O2 • the counterfactual is simply ‘O2’ • policy impact is ‘O1’- ‘O2’

  17. Random Assignment • In ERA, half of those willing to be randomly assigned were randomly assigned to the program group and half to a control group • Those assigned to the control group could not receive the services and financial incentives provided • The baseline information collected for ERA at the point of random assignment indicates that the two groups are very similar in terms of all observable measures • randomisation worked

  18. Random Assignment Continued • After entering ERA, employment, earnings, and other outcomes of the program group are compared for several years to those of the control group during the same years • Randomised evaluations of social programs used frequently in US (over 200 times) • It is increasingly used in other countries, especially developing countries; but so-far it has been used far less often than in the U.S.

  19. An Illustration • At random assignment about 25% of one of the ERA program groups worked • 24 months after random assignment, about 55% of the program group worked • Was this increase due to ERA?

  20. Illustration (continued) • 24 months after random assignment, about 52% of the control group worked • Thus, only about a 3 percentage point increase in employment (55% - 52%) is attributable to ERA

  21. Advantages of Random Assignment • As intervention and control groups were created randomly, they are statistically equivalent • Equivalent in both what we can observed about them and what we cannot • At follow-up, when me measure our outcome variables, the only difference between the two groups is the impact of the intervention • In theory provides unambiguous results • No need for complex statistics as with other methods - ease of interpretation • Baseline information not essential, but helpful

  22. Disadvantages of Random Assignment • On its own, only provides a measure of average impact - policy makers may have other questions about the policy • Can be expensive and complicated to implement • Sometimes impractical to implement • Possible lack of generalizability • Can create political problems by denying services to controls (but if resources are limited some method must be used to deny some) • In many cases, can take time for results to emerge • Other evaluation designs may also be subject to these limitations-- depends on what is being evaluated

  23. Conclusion • Trade off -- rigour against difficulty in implementation • Random assignment can be expensive - policy budget! • Experimental methods require lots of data • Only answers certain types of questions • Still, when it is feasible, random assignment provides the most accurate estimates of impacts

  24. Some Early Findings(* = statistically significant impact)

  25. Lessons from the ERA Design Work • Developing ownership among those fielding and running a program important • Randomized experiments are feasible, but circumstances must warrant their use • Use of multi-disciplinary teams should be encouraged • Designing programs and evaluations in tandem should be done whenever possible

More Related