Greg Mason, PRA Inc. March 23, 2007

1. Lab and Field Experiments in Economics:Survey and Assessment of their Potential for Policy Analysis Greg Mason, PRA Inc. March 23, 2007

2. Purpose of the paper • Present a high-level review lab and field experiments • Trace the origins of lab and field experiments in economics • Present the requirements for conducting experiments • Review recent experiments • Discuss how experiments may be used in policy development and evaluation.

3. The nature of experiments • Experiments can be exploratory or confirmatory • Exploratory experiments • Trial and error in manipulation of potential interventions creates a range of outcomes • Induction to infer potential hypotheses in relation to interventions and outcomes (cause and effect). • Confirmatory experiments • Test (confirm/disconfirm) a theory or hypothesis about causal relationships • Control all influences on outcomes, except the hypothesized cause • Process of deduction – inferring cause through the modeling of a controlled process.

4. Key elements of an experiment • Manipulation of selected and isolated components (treatments) in the physical and/or social environment; • Observation of changes in that environment before/after or with/without the manipulation; and • Control of any factor that could change the outcome except for the treatments.

5. Example 1: A simple lab experiment • A researcher offers the following deal to participants : • Participant A will receive $10 and can choose whether or not to offer part of that amount to Participant B. Participant B can choose whether or not to accept the offer. • Rationality and selfishness predict that A should only offer a minimal amount — say $1 — and that B will accept, since something is better than nothing. • When the game is played in the lab, B will reject any unfair offer, and A, anticipating this, offers more than a token minimum, typically $4. • As modern economic theory stresses, rational self-interest also includes altruism, especially if participants expect to play again.

6. Origins of experimental economics – 1 • Many economists have denied the possibility of conducting experiments “Economists cannot perform control experiments of chemists and biologists because they cannot easily control other factors.” Milton Friedman • Broadening the definition of experiment to include research that explores reactions of randomized groups to variations in policy and program design, opens an array of potential methods.

7. Origins of experimental economics – 2 • The first step in experimental economics followed the traditional lab experiment in psychology • Economics has moved from the lab to the field to increase: • Salience • External validity

8. The RCT trial - Donezepil

9. Randomization and statistical equivalence • Randomization into a treatment and control group creates two groups that are statistically equivalent: • For any statistic (mean, variance…) the two groups will return results that are the same (within bounds of statistical significance) • The test of statistical equivalence applies to observable and unobservable attributes.

10. Validity in experiments • Internal validity • supports the logical links between the objective of the policy or program, the design of the treatment, and the expected effect • External validity • extends the results from the experimental group to a wider population of interest • Treatment/dosage manipulation • confirms that outcomes vary in step with the treatments.

12. Thought experiments • A “what if” exercise that can be performed mentally, mathematically, or simulated using a spreadsheet • Starts from axioms of behaviour and possibly estimates of empirical relationships. • Economics starts a thought experiment with “let us assume….” • Theorists manipulate equations and model potential causal relationships.

13. Independent variables act as the “control” Cause and effect are imposed on the model. Rejection of a null does not “prove” causality. Multicollinearity and simultaneous equation bias complicate control. Ever more complex modelling to compensate. Observational study

14. Social experiment • Replicates the approach of the natural and medical scientist. • Often cited as the “gold standard” in economic policy evaluation. • Offers strong basis for inferring causal relationships • Logistically complex • Expensive to conduct • Control can be challenging (creating orthogonal treatments, attrition bias…)

15. Lab/Field Experiments • Economic lab experiments • use standard subject pools • abstract (somewhat artificial) settings – typically computer labs • a set of rules for the experiment. • Economic field experiments • Increase salience by creating real world analogues to the choice or decision • Use of financial rewards that are significant.

16. Two types of field experiment • Artefactual field experiments apply the lab experiment to non-standard subjects • Framed field experiments move the artefactual experiment into a field setting, such as a store, online purchasing, or choices made in a bank

17. Example 2: Job search options • Anecdotal evidence from LMDA evaluations suggests that counsellor attention is an important factor in motivating and orienting unemployed persons who are seeking work. • The effect of trained assistance on the speed with which EI claimants can generate job leads may be studied statistically or experimentally. • The statistical approach involves analyzing variation in the time spent generating job leads, using multivariate methods to understand the factors that affect outcomes. • An experimental approach would selectively withdraw and offer counselling assistance and observe the impact on the number of job leads.

18. Issues in experimental economics • theory validation • experimental control • internal and external validity • salience and coherence • financial incentives.

19. Theory validation • All theories require a set of untested assumptions or “maintained hypotheses.” It is impossible to control for everything. • The weakness in the falsification perspective is that, after rejecting a null hypothesis, an infinite number remain to be tested. • Successive experiments allow one to induce the validity of a causal relationship through the replication of studies and experiments.

20. Experimental control • Lab experiments tends to focus less on randomization subjects and more on the manipulation of alternatives to generate “data.” • Field experiments will randomize on some dimension of the alternatives. • The key is to create a “field” where the policy or intervention is subject to systematic control.

21. Internal and external validity • Internal validity increases in a lab or computer setting • The key to increasing internal validity is to remove confounding variables – i.e., isolate the subjects from the real world • External validity requires using subjects that are relevant to the policy question • A trade-off tends to exist between internal and external validity.

22. Salience and coherence • Salience ensures that participants see the experiment, and the options offered, as personally relevant • Coherence means that the alternatives offered to subjects represent a reasonably complete set for the policy.

23. Financial incentives Non-satiationmeans that subjects will always choose options with the greatest personal gain Saliency means that alternatives are important and relevant to the subject’s well-being Dominancemeans that the rewards of participation outweigh any costs (real and subjective) of participation Privacy means that the rewards are private to the individual.

24. The experiment: Approaching fair behaviour — Distributional and reciprocal preferences • Comparison of one-stage and multi-stage two-person games where individual decisions dictate the degree of equality and the level of personal gain among payoffs. • Hypothesis: one-stage games; decisions favouring equality over personal gain are motivated only by altruism. • In multi-stage games, where selfish decisions may be punished in subsequent rounds, decisions are motivated by altruism and feelings of reciprocity.

25. The experiment: Eliciting risk and timepreferences using field experiments • Modified version of a well-established laboratory model to a representative sample of the Danish population. • Experimental subjects face two types of choice decisions: • decisions between playing low- and high-risk lotteries with varying expected values. • decisions between small monetary payoffs now or larger monetary payoffs in the future. • Decisions favouring low-risk lotteries and immediate payments indicate risk aversion and a high level of time discounting.

26. The experiment: Saving decisions of the working poor — Short-and long-term horizons • Experimental subjects from the working poor who have three types of decision-making problems: • choosing between playing high- or low-risk lotteries with varying expected values. • deciding between small monetary payoffs immediately and larger payoffs in a number of weeks. • choosing between small monetary payoffs immediately and considerably larger payoffs in a number of years. • Comparison of results helped identify a stable relationship between short- and long-term time preferences among the working poor.

27. The experiment: The effects of educational vouchers on confidence — A field experiment to assess outcomes of educational policy • Presented to a random selection of applicant children, the voucher allotment actually created a natural experiment by establishing randomly selected control and treatment groups. • By comparing controlled confidence test results among lottery winners and losers, the authors established the net effect of the educational vouchers on child confidence.

28. The experiment: On the empirics of social capital • Based on Robber’s Cave experiment, where a group of homogeneous boys was separated into two groups, isolated, and asked to engage in a variety of team-building exercises. • When reintroduced to each other, experimenters tracked changes in the behavioural norms among the boys relative to those established prior to separation. Experimenters also noted the degree to which competition between groups facilitated collective group action.

29. The experiment: The targeted negative incometax in Germany • A negative tax incentive regime in Germany that was applied differentially to individuals within and across municipalities, to establish the regime’s success or failure. • The differential application of the regime within comparable regional settings established experimental treatment and control groups for further analysis. • Years after application, comparing the re-employment success of individuals under and not under the regime allowed researchers to establish the impacts of the negative tax incentive regime.

30. The experiment: Investment by working poorin human capital • Identify preferences for educational savings among the working poor by soliciting responses to a variety of choice questions regarding current consumption and savings for future education. • Experimental subjects are asked to choose between a small immediate monetary payoff and a larger monetary contribution to an educational saving fund meant to support future education or training. • Varying the level of immediate and educational saving payoffs allows experimenters to establish the level of incentive necessary to make individuals among the working poor invest in education.

31. The experiment: Discrete choice methodsfor estimating labour supply. • A client survey, conducted as part of the National Child Benefit evaluations, randomly assigned social assistance respondents to three groups. • Respondents in each group agreed or disagreed with a specific question: “Would you accept a job at $Y per hour?” where Y varies with the group. • The result is an experimentally determined reservation wage that measures the height of the welfare wall. • Covariates show how the reservation wage/welfare wall varies with the attributes of the respondent.

33. The experiment: Stated choice experiment insocial policy preferences • Focus group participants (NCB evaluation) rated various options for income and in-kind support. • A stated choice experiment consists of a trial, or a series of trials, typically conducted as part of a focus group, in which the researcher makes purposeful changes to the attributes/levels of a policy package in order to observe and identify possible explanations for changes in the response variable. • In the simplest form of analysis, the resulting data set is a regression model, where the valuations of packages become the dependent variable and the packages are coded using dummy variables.

35. Summary • Lab experiments have been effective in exploring theory and the assumption of human economic behaviour. • Field experiments increase salience and therefore provide a more secure policy development platform. • Randomization and experimentation are not synonymous – the key to field experiments is controlled manipulation. • A variety of field experimental formats are available to support cost-effective policy design and evaluation. • The most significant benefit of field experiments are cost and flexibility. • Field experiments can manipulate several dimensions and levels within the same context. • A weakness is that salience can be hard to achieve in any experimental setting – most experiments are not “real life,” and participants know this.

36. How HRSDC can use field experiments • Basic research into preferences and motivations for populations at risk. • Value exists in funding small scale field experiments with target population to determine service mix, client response to policy options, etc. Typical scenarios include: • Assessing counseling modalities for EI clients (by age, gender, ethnicity, etc) – what works best for whom. • Fine tuning policy options such as support levels for earnings supplements or earnings disregards. • Testing the response of parents to educational investments (i.e., testing the features of a RESP or other savings plan). • Developing a policy package for EI clients (i.e., revamping the EBSMs) by using conjoint methods where clients rate training features. Another approach is to find optimum policy packages within a budget constraint.

37. Final thoughts • Field experiments will not replace other methods, but serve as a low-cost complement to existing methods. • Since HRSDC (and other departmental evaluations) are undertaking surveys and focus groups, many experimental methods can be grafted onto the existing research platform at low marginal cost, without affecting the information obtained from these other lines of evidence. • Experiments introduce important insights that support policy design (and redesign) into evaluations, thereby providing increased value for money for the study. • Value still exists in the large scale experiments, which will often benefit from the smaller field experiments at the design phase.