Testing the Causal Effects of Social Capital: Design for a Cluster-Randomized Field Trial

1. Testing the Causal Effects of Social Capital: Design for a Cluster-Randomized Field Trial Adam Gamoran and Ruth N. López Turley University of Wisconsin-Madison

2. Social Capital: Conceptual and Causal Ambiguity • Social capital is one of the most popular terms in social science today • Viewed as the source of many positive outcomes • Test scores, school completion, social adjustment, mental and physical health • Decline of social capital is seen as responsible for many social ills • Crime, apathy • Causal role of social capital is ambiguous

3. Social Capital: Conceptual and Causal Ambiguity • Concept of social capital is also ambiguous • Relations of trust, mutual expectations, and shared values embedded in social networks • Not possessed by individuals • Resides in the relationships individuals have with one another • Individuals can draw upon social capital in their networks • Social capital facilitates the flow of information and the development and enforcement of norms

4. Concepts of Social Capital • Qualities of social networks that signify social capital: How do we know if social capital is present? • Intergenerational closure

5. Intergenerational Closure Source: Coleman, Am. J. Soc., 1988

6. Concepts of Social Capital • Qualities of social networks that signify social capital: How do we know if social capital is present? • Intergenerational closure • Trust • Network members rely on one another • Facilitates sharing of norms and information • Shared expectations • Also facilitates supporting norms and distributing information

7. Concepts of Social Capital • Contrary to Coleman (1988), we do not define social capital by its function • Contrary to Portes (1998), we view social capital as a collective rather than as an individual attribute • We follow Sampson et al. (1999): “…social capital for children refers to the resource potential of personal and organizational networks…”

8. Domains of Social Capital • Parent-school relationships • Parent-parent relationships • Parent-child relationships

9. Mechanisms of Social Capital • For young children, social capital operates through their parents • Two primary mechanisms • Social support • Parents who feel more connected to others have better access to information and are better able to establish and enforce norms with their children • Social control • Parents’ positive social networks offer collective socialization of children

10. Social Capital and Inequality • Unequal social capital contributes to unequal child development • Among U.S. Latinos, social capital within family networks is high, but parent-school social capital is low • Building family-school social capital may enhance child outcomes particularly for Latinos – the focus of our empirical analysis

11. The Causal Role of Social Capital • Many studies have tested the relation between social capital and child outcomes • Most rely on longitudinal data • Nonetheless, causal direction is ambiguous • Does social capital foster school success, or do stronger social ties emerge in communities that have more effective schools? • Some group norms can negatively affect child outcomes!

12. The Causal Role of Social Capital • Survey research may overestimate effects of social capital (Mouw, 2006) • Endogeneity: Group members influence one another at the same time • Unobserved selectivity: unmeasured conditions lead to both common memberships and common outcomes • Statistical efforts to resolve these causality issues rely on questionable assumptions • E.g., effects are unbiased net of control variables

13. The Causal Role of Social Capital • An experimental design offers a more rigorous approach to testing social capital’s causal role • No unobserved selectivity: Assignment to “treatment” is random • Avoid endogeneity problem through multilevel assessment of social capital effects

14. The Causal Role of Social Capital • Conditions for an experimental assessment of social capital effects: • An intervention that manipulates social capital • Random assignment to treatment and control • Random assignment of groups of individuals (because social capital is an attribute of groups, not individuals) • Tools for measuring social capital and outcomes • Statistical methods suitable for analysis of a cluster-randomized trial

15. 1. An Intervention that Manipulates Social Capital • FAST: Families and Schools Together • A multi-family group prevention program • Implemented in three stages • Outreach to parents • 8 weeks of multi-family group meetings • 2 years of monthly follow-up meetings led by parents

16. 1. An Intervention that Manipulates Social Capital • Elements of FAST • Led by a parent-professional partnership • Culturally representative and adapted • Research-based activities • Family meal • Group singing • Family games • Parent support/ children’s time • One-to-one responsive play • Closing circle

17. 1. An Intervention that Manipulates Social Capital • Prior research on FAST • 4 previous randomized trials have documented positive outcomes for children’s social and academic outcomes • These studies have occurred at the individual or classroom levels • School-wide, “multi-hub” FAST is likely to have even more powerful effects

18. 1. An Intervention that Manipulates Social Capital • Prior research on FAST • FAST builds social capital • Parent-school: Reduces alienation from school authorities, and increases comfort level • Parent-parent: Reduces isolation of parents by creating a parent support group • Parent-child: Improve relationship through one-on-one responsive play • Particularly valuable for immigrant communities

19. Conceptual Model

20. 2. Random Assignment to Treatment and Control: Experimental Design • Research Sites • San Antonio, TX: A large, long-standing Latino populations (51% of students) • Milwaukee, WI: A rapidly growing Latino population (21% of students) • Experience with FAST, community agencies available to implement • Agreed to implement FAST in treatment schools, not in control schools • Subject to agreement of principals and teachers • They love FAST, this won’t be a problem

21. 3. Random Assignment of Groups of Individuals: Experimental Design • 26 schools from each district (13 treatment and 13 control), total of 52 schools • All first-grade families will be invited to participate • We anticipate 75% participation rate, 20% attrition rate = 60% long-term follow-up • Three years of data collection (grades 1 to 3)

22. 3. Random Assignment of Groups of Individuals: Experimental Design • How did we decide on 52 schools? • Power analysis

23. Power Analysis: Assumptions • Power criterion (1 – β) = .80 • Probability of Type I error () = .05 • Within-school sample size (n) = 60 • Effect size () =.25 • Intraclass correlation () = .10 • Covariate correlation (r) = .40-.60

24. Power Analysis: Softwarehttp://sitemaker.umich.edu/group-based/optimal_design_software

25. Power Analysis

26. Power Analysis: Conclusion • Under reasonable assumptions, a sample of 52 schools will provide sufficient power to detect the effects of social capital, if they exist.

27. 4. Tools for measuring social capital and outcomes • Outcomes • Parent and teacher ratings of child social skills and problem behaviors (grades 1 and 3) • Teacher ratings of child academic competence • High-stakes standardized tests of reading and mathematics

28. 4. Tools for measuring social capital and outcomes • Social capital • Parent social capital questionnaire • Our only pre-intervention measure • Not really needed for experimental design, but of interest in its own right • Follow-up measures in the spring of grades 1 and 3 • Key sources: Bryk and Schneider (2002); McDonald and Moberg (2002)

29. Parent Social Capital Questionnaire • Parent-school trust, shared expectations

30. Parent Social Capital Questionnaire • Parent-parent closure, trust, shared expectations

31. Parent Social Capital Questionnaire • Parent-child trust, shared expectations

32. 4. Tools for measuring social capital and outcomes • Social capital • Parent Involvement in School Questionnaire • Indicators of trust and shared expectations in parent-school and parent-child relationships • Separate forms with parallel questions from parent and teacher perspectives • Source: Shumow, Vandell, and Kang (1996) • Completed by teachers and parents at the end of grades 1 and 3

33. 4. Tools to measure social capital and outcomes: Other variables as indicated

34. 5. Statistical methods suitable for analysis of cluster-randomized trial • This study relies on place-based random assignment • CRT: Cluster-randomized trial • Randomization is at the aggregate level • Well suited to contextual investigations • Must assess the intervention at the level at which randomization occurs

35. 5. Statistical methods suitable for analysis of cluster-randomized trial • A multilevel model is the appropriate statistical approach to analysis of CRT • Captures variability both at the level of the cluster and within clusters • In our case: students within schools • Treatment is at the level of the school • Theoretically, social capital is also at the level of the school • We allow for individual-level variation

36. 5. Statistical methods suitable for analysis of cluster-randomized trial • School-level control variables reduce variation between schools, permit more precise treatment effects • Individual-level background controls also increase precision • More importantly, multilevel interactions permit estimation of differential treatment effects

37. Multilevel Models: Linear Outcomes Level 1. Yij = ß0j + ß1j(SEX)ij + ß2j(LATINO)ij + ß3j(BLACK)ij + ß4j(POVERTY)ij + rij Level 2. ß0j = γ00 + γ01(MEAN PRIOR ACH)j+ γ02(PERCENT POVERTY)j+ γ03(PERCENT LATINO)j + γ04(PERCENT BLACK)j + γ05(FAST)j + γ06(CITY)j+ γ07(PERCENT LATINO x FAST)j + γ08(PERCENT BLACK x FAST)j + u0j Level 2. ß2j = γ20 + γ21(FAST)j + γ22(CITY)j + u2j ß3j = γ20 + γ21(FAST)j + γ22(CITY)j + u3j ß4j = γ20 + γ21(FAST)j + γ22(CITY)j + u4j

38. 5. Statistical methods suitable for analysis of cluster-randomized trial • By adding social capital to the model, we test whether social capital accounts for the effects of FAST on child outcomes • Main focus is on school-level effects

39. Multilevel Models: Linear Outcomes Level 1. Yij = ß0j+ ß1j(SEX)ij + ß2j(LATINO)ij + ß3j(BLACK)ij + ß4j(POVERTY)ij + ß5j(SOCIAL CAPITAL)ij + rij Level 2. ß0j = γ00 + γ01(MEAN PRIOR ACH)j + γ02(PERCENT POVERTY)j + γ03(PERCENT LATINO)j + γ04(FAST)j + γ05(MEAN SOCIAL CAPITAL)j + γ06(CITY)j + u0j

40. 5. Statistical methods suitable for analysis of cluster-randomized trial • Additional challenges • Uncommon measures: Different tests in Texas and Wisconsin • Linking strategy, corrected for unreliability • Examine probability of reaching the proficiency threshold rather than test score

41. 5. Statistical methods suitable for analysis of cluster-randomized trial • Additional challenges • Bias in social capital effects • FAST effects will be estimated without selectivity bias • Social capital effects will also be estimated without selectivity bias if they derive only from FAST • This is probably not the case • If social capital occurs independently of FAST, an omitted variable may affect social capital and child outcomes • Use pre-FAST measure to check • Use FAST as an instrument for social capital • Control for pre-FAST social capital

42. 5. Statistical methods suitable for analysis of cluster-randomized trial • Additional challenges • Bias in social capital effects • Differential non-response by treatment and control parents • Consent will be obtained prior to randomization • Follow up a random subsample of non-respondents with home visits

43. 5. Statistical methods suitable for analysis of cluster-randomized trial • Additional challenges • Fidelity of implementation • Implementation study • Implementation checklist • Interviews, focus groups with parents and teachers • Including interviews with 2 non-participating parents in each treatment school • Qualitative data will provide more nuanced insights on the mechanisms through which FAST affects (or does not affect) child outcomes

44. Conclusions • The term “social capital” has reflected many different ideas in different writings • Causal ambiguity has been a consistent limitation of social capital research • By manipulating social capital experimentally, we aim to provide a more persuasive test of social capital effects

45. References • Bryk, A. S., & Schneider, B. L. (2002). Trust in schools: A core resource for improvement. New York: Russell Sage Foundation. • Coleman, J. S. (1988). Social capital in the creation of human capital. American Journal of Sociology, 94(Suppl.), S95–S120. • McDonald, L., & Moberg, D. P. (2002). Social relationships questionnaire. Madison, WI: FAST National Training and Evaluation Center. • Mouw, T. (2006). Estimating the causal effects of social capital: A review of recent research. Annual Review of Sociology, 32, 79–102. • Portes, A. (1998). Social capital: Its origins and applications in modern sociology. Annual Review of Sociology, 24, 1–24. • Sampson, R. J., Morenoff, J. D., & Earls, F. (1999). Beyond social capital: Spatial dynamics of collective efficacy for children. American Sociological Review, 64(5), 633–660. • Shumow, L., Vandell, D. L., & Kang, K. (1996). School choice, family characteristics, and home-school relations: Contributors to school achievement? Journal of Educational Psychology, 88, 451–460.

46. Further Reading on Cluster-Randomized Trials • Bloom, H. S. (2006). Learning more from social experiments: Evolving analytic approaches. New York: Russell Sage Foundation. • Bloom, H. S., Bos, J. M., & Lee, S. W. (1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23, 445–469. • Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N., & Chambers, B. (2005). Success for All: First-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27(1), 1–22. • Boruch, R., May, H., Turner, H., Lavenberg, J., Petrosino, A., & de Moya, D. (2004). Estimating the effects of interventions that are deployed in many places: Place-randomized trials. American Behavioral Scientist, 47, 608–633. • Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.

47. Further Reading on FAST • Abt Associates. (2001). National evaluation of family support programs: Vol. B. Research studies: Final report. Cambridge, MA: Author. Retrieved February 12, 2007, from http://www.abtassoc.com/reports/NEFSP-VolB.pdf • Kratochwill, T. R., McDonald, L., Levin, J. R., Young Bear-Tibbetts, H., & Demaray, M. K. (2004). Families and Schools Together: An experimental analysis of a parent-mediated multi-family group intervention program for American Indian children. Journal of School Psychology, 42, 359–383. • McDonald, L., Moberg, D. P., Brown, R., Rodriguez-Espiricueta, I., Flores, N., Burke, M. P., et al. (2006). After-school multifamily groups: A randomized controlled trial involving low-income, urban, Latino children. Children and Schools, 18, 25–34. • U.S. Office of Juvenile Justice and Delinquency Prevention. (2006). Families and Schools Together (FAST). In U.S. Office of Juvenile Justice and Delinquency Prevention, OJJDP model programs guide. Retrieved February 11, 2007, from http://www.dsgonline.com/mpg2.5/TitleV_MPG_Table_Ind_Rec.asp?ID=459 • U.S. Substance Abuse and Mental Health Services Administration. (2005). Families and Schools Together (FAST). In U.S. Substance Abuse and Mental Health Services Administration, SAMHSA model programs: Effective substance abuse and mental health programs for every community. Washington, DC: Author. Retrieved February 11, 2007, from http://www.modelprograms.samhsa.gov/pdfs/Details/FAST.pdf