Lab Experiments for Measurement in Program Evaluation

1. Lab Experiments for Measurement in Program Evaluation Michael J. Gilligan, New York University

2. The Task • Government/NGO/CBO programs wish to change participants attitudes and beliefs in particular ways • Typically these program coach participants in the ‘right’ set of attitudes and beliefs. • Examples • Pro-social behaviors: contributions to public goods, trust, tolerance, non-violence and so on • Attitude and behaviors toward marginalized groups: women minorities, particular ethnic groups • These programs would like to be able to measure whether their efforts have been successful

3. The Problem • Randomized control trials are essential to be able to make causal statements about the effects of the program • But randomized control trials are not a solution to the measurement problem—indeed they are a hindrance to it. • RCT programmers only operate with ‘treated’ populations so only treated populations receive coaching on the ‘right’ responses • RCTs, the very thing that is insuring unbiasedness with respect to subject pools (balance) is introducing bias in measurement

4. Social Capital & Pro-social Attitudes

5. Definition • [S]ocialnetworks and the norms of reciprocity and trustworthiness that arise from them. …[S]ocialcapital is closely related to ... “civic virtue.'' The difference is … civic virtue is more powerful when embedded in a dense network of reciprocal social relations. A society of many virtuous but isolated individuals is not necessarily rich in social capital (Putnam 2000).

6. We are interested in measuring: • Altruism • Trust • Trustworthiness • Willingness to contribute to public goods • The social networks that (purportedly) support these behaviors

7. Implications for Development • Trust: crucial for cost-effective self enforcement of contracts • Compliance with social norms: non-violence, compromise, fairness • Contributions to public goods: essential for economic efficiency • Respect for legitimate sources of authority

8. A Few Findings (among many) • Putnam (1993) shows that local governments in Italy are more efficient where there is greater civic engagement. • Knack and Keefer (1997) demonstrate that increases in country-level trust lead to large increases in the country’s economic growth. • La Porta et. al. (1997) establish a strong positive link between trust and judicial efficiency and a strong negative link between trust and corruption.

9. Implications • The World Bank and other international actors have many programs to foster social capital and pro-sociality • Community-based DDR • Community-driven development programs • A focus on local capacity in development efforts • “Local ownership” of development programs to foster sustainability

10. Measuring Social Capital and Social Norms • These are very difficult concepts to measure • In many cases they are not observed directly • Indicators differ greatly across different cultures • People are often unwilling to reveal behavior that is not pro-social

11. Traditional survey measures • ‘Generally speaking, would you say that most people can be trusted or that you can’t be too careful in dealing with people?’ (World Values Survey) • “Would you be willing to contribute a day of free time to … ?” • How difficult do you think it would be for your community to reach agreement on …?” • In the last three months have you contributed time or money to a community-based organization? • Did you vote in the last election?

12. Bias concerns with surveys • Programmers coach respondents in the ‘right’ answers to these types of questions • They do not operate in control communities at all so respondents many not even know the ‘right’ answers

13. Observational Measures • Number of people who voted in the last election • Number of people who show up to clean up a public park • Contributions to a community fund

14. The measures have great external (real world) validity but … • Are we measuring social attitudes or leadership strength? • … or intimidation? • …or corruption? • Example: Voter turnout in the Soviet Union was routinely above 98 percent • ‘Good’ outcomes may be caused by the exact opposite of good institutions and pro-social attitudes

15. Structured Observational Measures • ‘Structured Community Activities’ (Casey Glennerster and Miguel) • Funds collected in matching-grant scheme • Decision making over allocating salt or batteries • Allocation of tarpaulin • Tuungame Project, Congo (Humpreys, Sanchez de la Sierra and van der Windt 2013) • Participation in matching funds for a public good • Allocation of a $100 ‘windfall’ • Participation in a community meeting

16. Structured Observational Measures • Structured and therefore more comparable to each other • Have great external validity … • but we still cannot disentangle individual factors (attitudes) from community-wide factors (leadership, institutions)

17. Lab-in-the Field Activities • Observing behavior in a controlled laboratory setting • All social pressures, political institutional effects etc., are removed by design of the experiment • We observe only people’s responses to the incentives that we (the experimenters) offer them • We are able to disentangle attitudes from community-wide factors

18. Loss in External Validity • Community-wide factors (leadership, institutional efficiency) are excluded from the lab so we cannot obtain measures of them • Thus lab activities are best combined with the other measurement methods

19. Behavioral games • Three important games are: • Altruism game • Trust game • Public goods game • Our main interest is in the altruism, trust and public goods games, but we also need to conduct the other games to control for risk attitudes, patience and altruism

20. Game Instruction

22. Altruism Activity • Subjects were given a sum of money • Nepal; 40 NPR in 5 NPR notes • Sudan: 3 pounds in half-pound coins • Cambodia: 16,000 KHR in 4,000 KHR notes • Subjects decide how much they want to contribute to a local needy family • The identity of the family is not revealed

24. Trust/Trustworthiness Activity • Subjects are randomly assigned to one of two roles: sender or receiver (we use neutral names in the field) • Both types are given initial endowment of money • Senders decide how much of their endowment to send to the receiver • We triple that amount and give it to the receiver • The receiver decides how much of this total to return to the sender • All players and types are anonymous • Nash: send zero, return zero • Social optimum: send full endowment, return whatever is necessary to support trusting behavior

26. Public Goods Game • All subjects play simultaneously • Each player is given two cards, one with an “X” and one blank • For each “X” card turned in in the first round all players receive an amount of money, say 4NPR • Turning in an “X” card in the second round earns the player that turned it in a larger amount, say 20 NPR

27. Attitudes Toward Marginalized Groups

28. Examples • Many programs are interested improving the status of marginalized groups, especially women • Governments/NGOs/CBOs are often interested in easing (often violent) ethnic rivalries, especially in post-conflict settings

29. Same Problem • RCT programmers only operate with ‘treated’ populations so only treated populations receive coaching on the ‘right’ responses • RCTs, the very thing that is insuring unbiasedness with respect to subject pools (balance) is introducing bias in measurement

30. A Variety of Options • Standard games (altruism, trust, public goods etc.) can be used to measure attitudes toward ‘out groups’ groups • Bracic 2013 attitudes toward Roma in the former Yugoslavia • Observing behavior of deliberation, cooperation and teamwork among mixed groups • Karpowitz and Mandelberg 2014 deliberation in mixed groups of men and women

31. Observing group behavior • Bales Interaction Process Analysis • Participants are given a task that requires a group decision or cooperation • Record interactions according to a specific set of criteria to code whatever the researcher is interested in measuring (respect, hostility, etc.) • The trick • Not cuing participants that this is a study of in-group out-group interaction • Incentivizing participants to act according to beliefs about the out-group

32. Example: Attitudes toward Gender and Ethnicity in the Liberian National Police (LNP) • The government of Liberia adopted an explicit 30% quota for women in the LNP • We did NOT conduct an RCT but we were interested in: testing some of the assumptions of the gender program

33. Program proponents claimed that more women would produce a variety of benefits • More consensual decision making • Greater sensitivity to gendered crimes • Decades of social psychology findings that women would not participate fully in group deliberations.

34. The program had been underway for several years so officers new the attitudes toward female officer that they were supposed to have • Thus a survey would not have been a convincing measurement strategy • We had groups of size officers complete team tasks and randomized the number of female officers in each group • We observed team members’ to see • if men reacted differently in groups with more women • Groups with more women deliberated more consensually and were more likely to see crime as gendered

35. Findings • Female officers were not, in general, more likely to see a gendered crime but more competent women were • Groups with more women members were not more likely to see a gendered crime • Groups with more women were not more consensual • Backlash effect: Men in majority female groups were significantly more aggressive.

36. Conclusion • Programming by its very nature coaches beneficiaries in giving the types of survey responses answers the program would like to hear • Randomization exacerbates this problem • Behavioral measures are appealing but: • Measures with high external validity can make it hard to disentangle mechanisms at the individual and community level • Fine tuning individual incentives correctly get at attitudes even when subjects are cued to the ‘right’ answer: monetary reward will induce people will act on actually held beliefs rather than the ‘socially correct’ ones • Lab-in-the field activities address both of these issues and provide an important tool for measuring the social effects of programs, at some loss of external validty