Summary of Themes

1. Evidence Based Policy, Evidence Grading Schemes and Entities, and Ethics in Complex Research Systems Robert Boruch, University of Pennsylvania September 14-15 2008 5th European Conference on Complex Systems Jerusalem

2. Summary of Themes • People want to know “what works” so as to inform their decisions. • The scientific quality of evidence on “what works” is variable and often poor. • People’s access to information on what works through the internet is substantial. • Organizations and data bases have been created to (a) develop evidence grading schemes, (b) apply the schemes in systematic reviews of evidence from multiple studies, and (c) disseminate results through the internet.

3. The “What Works” Theme • “What works” refers here to estimating the effects of social, educational, or criminological interventions • In a statistically/scientifically unbiased way • And so as to generate a statistical statement of one’s confidence in the results.

4. Evidence Based Policy/Law: A Driver of Interest in What Works • US, Canada, UK, Israel (e.g. National Academy) • Sweden, Norway, Denmark • Australia, Malaysia, China • Mexico, others in Central America • Multinationals: OECD, World Bank • Others

5. Information Glut as Driver: Naïve Web Searches A Google search on “evidence based, ” yields 9, 660,000 links. A Google search on “what works” yields 6,350,000 links (.21 seconds). A Google search on “evidence based practice” yields 2,000,000 links (.42 seconds). A Google search on “evidence based policy” yields 132,000 links (.35 seconds). • What are we to make of this?

6. Publication Rates in Education • 20,000 articles on education published each year in English language journals • 2/1,000-5/1,000/year report on controlled trials of programs, policies, or practices to estimate effectiveness • For every curriculum package that has been tested in a controlled trial, there are 50-80 that are claimed to be effective based on no defensible scientific evidence.

7. Relevant Organizations Nested • National, State/Provincial, Municipal: Policy or law • Agencies with nation, etc., e.g. National Science Foundations, Institute for Education Sciences (US), University Research • Programs and projects within agencies • Data bases and reports within projects • Users of information at each level, e.g. scientists, policy people, the public

8. International Organizations: NGOs • Cochrane Collaboration in Health Care: http://cochrane.org • Campbell Collaboration in education, welfare, crime and justice: http://campbellcollboration.org

9. Two Examples Here • International Campbell Collaboration in education, welfare, crime and justice • What Works Clearinghouse in education (Institute for Education Sciences, US)

10. Data Bases in this Context • Evidence Grading Schemes currently focus on reports of statistical analyses of impact, not micro-records of individuals as yet. • Example: 5-10 statistical reports (ingredients of part of data base) on evaluating impact of conditional income transfer programs in developing regions • Example: Cochrane Collaboration data base on randomized trials contains nearly .5 million such reports • “Meta-analysis” of results of multiple studies

11. C2 SPECTR • C2 Social, Psychological, Educational, and Criminological Trials Register • 13,000+ entries on randomized and possibly randomized trials • Feeding into C2 systematic reviews • Feeding into the IES What Works Clearinghouse (USDE)

12. The Campbell Collaboration • Mission: since 2000, prepare, maintain and make accessible C2 systematic reviews of evidence on the the effects of interventions (“what works” ) to inform decision makers and stakeholders. • International and Multidisciplinary: Education, social welfare/services, crime and justice • http://campbellcollaboration .org • Precedent: Cochrane Collaboration in health (1993)

13. 1. Collaborating across Nations and Disciplines 2. Building on Enthusiasm 3. Avoiding Duplication 4. Minimizing Bias 5. Keeping Current 6. Striving for Relevance 7. Promoting Access 8. Ensuring Quality 9. Maintaining Continuity Nine Key Principles of C2: A Scientific Ethic

14. What are Evidence Grading Schemes (EGSs) ? • These are inventories (guidance, checklists, scales) or processes that… • facilitate making transparent and uniform scientific judgments about… • The quality of evidence on effects of programs or practices or policies

15. C2’s and Others’ Major Evidence Grading Distinction on What Works • Randomized controlled trials yield the least biased and least equivocal evidence on “what works” i.e. effect of a new intervention (program, practice, etc.) • Alternative methods to estimate the effect of interventions yield more equivocal and more biased estimates of effect, e.g. “before-after” evaluations and other nonrandomized trials. • Both randomized trials and nonrandomized trials are important, but they must be separated in evidence grading schemes.

16. Example: Randomized Controlled Trial • Individuals or entities such as villages or organizations are randomly allocated to one of two or more interventions • The random allocation assures a fair comparison of the effects of the interventions • And the random allocation assures a statistically credible statement about confidence in the result, e.g. confidence interval and statistical tests

17. More Specific Example • A new commercially curriculum package for math education is the intervention under investigation • The new curriculum is RANDOMLY allocated to half of a sample of 100 schools, with the remaining half of schools serving as a control group, so as to form two equivalent groups of schools (fair comparison) • The outcomes, such as achievement test scores, from the intervention group and the control group are compared

18. Entities and Evidence Grading Schemes for What Works • Cochrane Collaboration: Systematic reviews in health • Campbell Collaboration: crime, education, welfare • Society for Prevention Research (Prevention Science 2006) • What Works Clearinghouse, Institute for Education Sciences WWC IES http://whatworks.ed.gov • Food and Drug Administration, other regulatory agencies • National Register of Evidence-based Programs and Practices • Others: California etc.

19. What are the Ingredients of EGSs?

20. WWC Aims • To be a trusted source of scientific evidence on what works, what does not, and on where evidence is absent… • Not to endorse products • http://www.whatworks.ed.gov

21. What Works Clearinghouse Illustration

23. Example: C2 Parental Involvement Trials • 500 possibly relevant studies of impact • 45 Possible Randomized Controlled Trials (RCTs) • 20 RCTs Met Study Inclusion Criteria18 RCTs included in the Meta-Analysis • Nye, Turner, Schwartz http//:campbellcollaboration.org

25. Example: Petrosino et al on Scared Straight Trials • Over 600 articles that are possibly relevant to impact of Scared Straight • Only 15 reach a “reasonable” level of scientific standard • Only 7 reached standard of being randomized controlled trial.

26. Figure 1. The effects of Scared Straight and other juvenile awareness programs on juvenile delinquency: random effects model, “first effect,” reported in the study (Petrosino, Turpin-Petrosino, and Buehler, 2002) n=number of failures N=number of participants CI=confidence intervals Random=random effects model assumed

27. Observational Studies Ashcroft: -50% crime Buckner: 0% Berry: -5% Mitchell -53% Several dozen others Randomized Trials Mich: +26% crime Gtr Egypt: +5% Yarb: +1% Orchow: +2% Vreeland: +11% Finckenauer: +30% Lewis: +14% C2 Product: Scared StraightPro Humanitate Award

28. Scientific Ethic • Providing access to scientific reports of evaluations of the effect of interventions, e.g. journal publications and limited circulation reports from governments or private organizations • Providing information beyond reports to assure understanding • In principle, but not always in practice, providing access to micro-records from impact evaluations

29. Ethics of Research on Humans • Evidence Grading Schemes and organizations need not worry about individual privacy because they have not access, as yet, to individuals records in identifiable form • They rely only on statistical/scientific reports that are published in peer reviewed journals and other reports and which include no individual records.

30. Ethics and Law: US • Individual rights to privacy are routinely assured on account of professional ethics statements and laws in the US. • The relevant codes of professional ethics in US include those of AERA, ASA, AAPOR, APA, and others. • The relevant laws in the US include Family Education Rights and Privacy Act (FERPA), Privacy Act, HIPPA

31. Ethics and Randomized Controlled Trials • Relevant codes and law concern individual privacy and confidentiality of individual’s identifiable micro-records • Relevant regulations and codes include attention to informed consent (45CFR46) • Access to anonymous micro-records for secondary analysis is problematic and possibly unnecessary in this context

32. Appendices

33. Robert Boruch: Bio Boruch is the University Trustee Chair Professor in the Graduate School of Education and the Statistics Department of the Wharton School at the University of Pennsylvania, Philadelphia Pennsylvania Boruch is Fellow of the American Statistical Association, Academy of Experimental Criminology, American Academy of Arts and Sciences, American Educational Research Association Email: robertb@gse.upenn.edu

34. Provision to Advance Rigorous Evaluationsin Legislation • The program shall allocate X% of program funds [or $Y million] to evaluate the effectiveness of funded projects using a methodology that – • Includes, to the maximum extent feasible, random assignment of program participants (or entities working with such persons) to intervention and control groups; and • Generates evidence on which program approaches and strategies are most effective. • The program shall require program grantees, as a condition of grant award, participate in such evaluations if asked, including the random assignment.

35. Provision to Advance Replication ofResearch-Proven Interventions • Agency shall establish a competitive grant program focused on scaling up research-proven models • Grant applicants shall – • Identify the research-proven model they will implement, including supporting evidence (well-designed RCTs showing sizeable, sustained effects on important outcomes); • Provide a plan to adhere closely to key elements of the the model; and • Obtain sizeable matching funds from other sources, especially large formula grant programs.

36. A Focus on Data Bases that Concern “What Works” • Here, the focus is on projects that generate evidence about “what works,” and what does not work using good scientific standards • This is different from a focus on projects or programs that generate information on nature of a problem, monitoring program compliance with law, etc.

37. What are the Campbell Collaboration (C2) Assumptions? • Public interest in evidence based policy and practice will increase. • Scientific and government interest in cumulation and synthesis of evidence on “what works” will increase. • Access to information and evidence of dubious quality and need to screen for quality of evidence will increase. • The use of randomized controlled trials to generate trustworthy evidence on what works will increase.

38. What are the Products? • Registries of C2 Systematic Reviews of the effects of interventions (C2-RIPE) • Registries of reports of randomized trials and non- randomized trials, (C2-SPECTR) and future reports of randomized trials (C2-PROT) • Standards of evidence for conducting C2 Systematic reviews • Annual Campbell Colloquia • Training for producing reviews • New technologies and methodologies • Web site: http://www.campbellcollaboration.org

39. What are Other C2 Products? • C2 Trials Register (C2 SPECTR): 13,000 entries • Annals of the American Academy of Political and Social Sciences: Special Issues • C2 Prospective Trials Register • C2 Policy Briefs • Annual and Intermediate Meetings: London, Philadelphia, Stockholm, Lisbon, Paris, Oslo, Copenhagen, Helsinki, Los Angeles

40. Hand Search vs Machine Based Search • Journal of Educational Psychology (‘03-”06) • Hand search: RCT=66 • Full Text Elec N=99: 59% accurate, 41% false positives, 24% false negatives • Abstract only Elect N=11: 91% accurate. 9% false positive, 85% false negative

41. What Is the Value Added ? • Building a cumulative knowledge base • Developing exhaustive searches • Producing transparent and uniform standards of evidence • International scope • Periodic updating • Making reviews accessible

42. C2 Futures/Tensions • C2 Production: AIR and others • C2 Publications v journals • C2 and governments and C2 apart from governments • C2 and Sustainability, C2 as voluntary Organization versus C2 and Spin Off Organizations and Products

43. What are Other Illustrative Reviews? • “Scared Straight” Programs (Done, Award) • Multi-systemic Therapy (Done) • Parental Involvement (Done) • After School programs (Due 12/05) • Peer Assisted Learning • Counter Terrorism Strategies (Under revision) • Reducing Illegal Firearms Possession