My Role in ERA

1. Designing a Random Assignment Social Experiment In the U.K.; The Employment Retention and Advancement Demonstration (ERA)

2. My Role in ERA • A member of the 6-person team in the British Cabinet Office that: • Designed the ERA program • Designed the evaluation of ERA to determine how effective it is • A member of the team evaluating ERA • I’m in charge of the cost-benefit analysis

3. My Talk Today • How it was designed • What the ERA program is • The ERA evaluation and how experimental methods are being used to evaluated it • Some early results • Some lessons from designing ERA (if time permits)

4. Employment Retention and Advancement Demonstration (ERA) • Design Work in 2001-2002 • Run as pilot program in 2003-2007 in 6 sites • Analysis conducted: 2007-2010

5. Why Was ERA Undertaken? • To test a program that tries to keep low-wage workers employed after they find jobs and help them advance • To promote the use of random assignment experiments in the U.K. • Because it was expected the value of the information obtained about the program would exceed the cost of the obtaining the information (“the rational paradigm) • This requires that the information is actually used

6. Program Features • Continued contact with Jobcentre Plus advisors (ASAs) after obtaining employment • Retention bonus of £400 every 17 weeks if works full-time for 13 weeks • Training bonus of £8 per hour of training • Must contact ASA to receive bonus • ASAs advise participants on job advancement

7. Unique Features of ERA • Design of Program and Evaluation done simultaneously • Design work done at British Cabinet Office • Expectation that decision on national implementation of pilot would be based on evaluation findings • Pilot run as large-scale random assignment experiment • Focus is on what happens after a job is obtained

8. Evaluation Components • Process or implementation study • Impact analysis • Cost study • Cost-benefit analysis

9. Project Characteristics • Long planning period • Absence of political pressure • Developing the program and evaluation designs in tandem • Allowed for a good evaluation design • Designing the project in the U.K. Cabinet Office • Permitted project team to focus on the project • Need to transfer project to DWP once designed • Random assignment

10. How Does Random Assignment Work? • Estimate the impact of a policy, a change in a program, or a intervention • Provide evidence of whether the policy has led to (or caused) the change it was designed to - a causal link! • The overall objective - to provide policymakers with evidence of whether their policy works

11. Measuring Impact • Experiments measure the impact of policies/interventions in terms of their impact on outcomes; • E.g. Does ERA increase earnings? • The outcome measure is the earnings of the program group

12. Establishing Causality • To establish that a policy or intervention has caused change to occur rather than some other factor • E.g. many factors will affect individuals earnings in addition to participation in ERA • If we find that earnings have increased among the program group, how do we rule-out the influence of other factors? • E.g. most persons who entered ERA were not employed, but some would inevitably have found jobs without ERA. • We do this by estimating what we call the ‘counterfactual’—what would have happened without the program

13. The Counterfactual • Is what would have occurred in the absence of the policy or intervention • E.g. what would have happened to earnings over the same period of time for the same individuals had ERA not existed? • This is unobservable or missing information • We have to estimate the counterfactual - that is determine what would have happened in the absence of ERA

14. Estimating the Counterfactual • Wide range of ways to do this • These vary in complexity, rigor and the degree of control required by those planning the evaluation • We will look at only one method: The simple two group random assignment experiment • When feasible, random assignment is the best method for estimating a counterfactual

15. Random Assignment • In theory the most rigorous way to assess the impact of a policy or intervention • Provides unbiased estimates of intervention impacts • How does it work? • You identify individuals or groups who are eligible for a new intervention or policy • You create two groups at random - intervention and control groups (essentially a computer flips a coin) • Intervention group receives the new service or intervention, control group does not.

16. Simple Randomised Experiment Intervention R Eligible population Intervention group Outcome = O1 Control group Outcome = O2 • the counterfactual is simply ‘O2’ • policy impact is ‘O1’- ‘O2’

17. Random Assignment • In ERA, half of those willing to be randomly assigned were randomly assigned to the program group and half to a control group • Those assigned to the control group could not receive the services and financial incentives provided • The baseline information collected for ERA at the point of random assignment indicates that the two groups are very similar in terms of all observable measures • randomisation worked

18. Random Assignment Continued • After entering ERA, employment, earnings, and other outcomes of the program group are compared for several years to those of the control group during the same years • Randomised evaluations of social programs used frequently in US (over 200 times) • It is increasingly used in other countries, especially developing countries; but so-far it has been used far less often than in the U.S.

19. An Illustration • At random assignment about 25% of one of the ERA program groups worked • 24 months after random assignment, about 55% of the program group worked • Was this increase due to ERA?

20. Illustration (continued) • 24 months after random assignment, about 52% of the control group worked • Thus, only about a 3 percentage point increase in employment (55% - 52%) is attributable to ERA

21. Advantages of Random Assignment • As intervention and control groups were created randomly, they are statistically equivalent • Equivalent in both what we can observed about them and what we cannot • At follow-up, when me measure our outcome variables, the only difference between the two groups is the impact of the intervention • In theory provides unambiguous results • No need for complex statistics as with other methods - ease of interpretation • Baseline information not essential, but helpful

22. Disadvantages of Random Assignment • On its own, only provides a measure of average impact - policy makers may have other questions about the policy • Can be expensive and complicated to implement • Sometimes impractical to implement • Possible lack of generalizability • Can create political problems by denying services to controls (but if resources are limited some method must be used to deny some) • In many cases, can take time for results to emerge • Other evaluation designs may also be subject to these limitations-- depends on what is being evaluated

23. Conclusion • Trade off -- rigour against difficulty in implementation • Random assignment can be expensive - policy budget! • Experimental methods require lots of data • Only answers certain types of questions • Still, when it is feasible, random assignment provides the most accurate estimates of impacts

24. Some Early Findings(* = statistically significant impact)

25. Lessons from the ERA Design Work • Developing ownership among those fielding and running a program important • Randomized experiments are feasible, but circumstances must warrant their use • Use of multi-disciplinary teams should be encouraged • Designing programs and evaluations in tandem should be done whenever possible