What Works Clearinghouse

1. What Works Clearinghouse Susan Sanchez Institute of Education Sciences

2. Purpose: To promote education decision making through a web-based dissemination system featuring comprehensive, systematic, high-quality reviews of studies on the effectiveness of educational interventions (programs, products, practices and policies).

3. The WWC Does Not: • Endorse educational interventions • Conduct field studies of the effects of interventions Rather, the WWC reports on the effectiveness of educational interventions as measured by available evidence.

5. Challenges • Research literature too abundant for individual selection. Where to begin? • Difficult to sift and know what to trust and what to use. Lots of claims of effectiveness out there, but what can you trust? • It has to be transparent if it’s to be believed. • Complex, technical issues can take more than forever and ever to resolve.

6. Challenges • Opps, we got the answer but now there are new advances in methodology. • Consumers need a fast response for evidence on what works. • Pretty wonky stuff. How do you create user-friendly web-based reports and products that meets your customers’ needs?

7. WWC Systematic Review Process A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to extract and analyze data from the studies that are included in the review.

8. WWC Systematic Review Process • Select the Topic • Develop the review protocol • Research questions • Search parameters • Inclusion/Exclusion criteria • Conduct a literature search & identify the research • Screen and code studies for relevance and methodological quality • Analyze the results of eligible studies • Summarize results

9. Study Review Process Develop Protocol Develop Protocol Submissions from the Public & Intervention Developers Literature Searches Screening Standards Does Not Pass Screen Pass Screen Study Reviewed Against the WWC Evidence Standards Meets WWC Evidence Standards • Meets WWC Evidence Standards •Meets WWC Evidence Standards with Reservations Does Not Meet WWC Evidence Screens

10. Three Stages of Review • Screen studies for relevance • Select only relevant evidence • Assessing strength of the evidence and sort by rigor • Select only credible evidence • Sort credible evidence by whether any reservations • Identify other important study characteristics

11. Stage 1: Screening for Relevance • All potentially relevant studies identified through the extensive search • Does the study provide evidence that, if credible, would be a valuable addition to the knowledge base regarding the effectiveness of the focal intervention?

12. Six Essential Screeners 1) Relevant time frame 2) Relevant intervention 3) Relevant sample 4) Relevant outcome 5) Adequate outcome measures 6) Adequate reporting

13. Stage 2: Assessing Strength of Evidence • The goal is to apply consistent criteria to sort evidence by its credibility into three buckets: Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Screens

14. The WWC Evidence Standards (applied to individual studies) • Meets Evidence Standards • RCTs without severe design or implementation flaws • Meets Evidence Standards with Reservations • RCTs with severe design or implementation flaws • QEDs with equating and without severe design or implementation flaws • Does Not Meet Evidence Screens • Studies not relevant to review • Studies with fatal design or implementation flaws

15. Five Factors Govern Judgments • Method for forming intervention and comparison groups • Evidence on baseline equivalence • Sample attrition • Possible contamination of study conditions • Threat of teacher confound

16. Stage 3: Other Study Characteristics • Variations in people, settings and outcomes represented • How generalizable are the findings? • Results reported by subgroups, settings and outcomes • What is the breadth of the outcomes reported? • Statistical results available • How complete is the reporting? • Do the estimates reflect statistical controls for baseline characteristics?

18. Evidence Base of Character Education Programs Over 70 programs were submitted to or identified by the WWC. 41 school-based programs met WWC definition and were eligible for review. 93 studies on the 41 programs were collected. 13 programs had at least one study meeting evidence standards, either with or without reservations 27 programs had no studies passing evidence screens 1 program is under review

19. 13 Character Education Programs with at Least one Study Meeting Evidence Standards, either with or without Reservations

20. Distribution of 92 Studies on 40 Programs across Evidence Categories Meet evidence standards Meet evidence standards with reservations Do not meet evidence screens

21. Common Reasons for Studies Failing to Meet Evidence Screens Not meet relevance screens Lack of a comparison group Severe overall or differential attrition for QEDs Lack of baseline equivalence for QEDs Confound in one-teacher/school-per-condition studies Inadequate statistical reporting for ES computation

22. For More Information on the WWC • Visit our website at www.whatworks.ed.gov • Subscribe for WWCUpdate, the WWC’s electronic news alert through our website • Phone: 1-866-WWC-9799 • E-mail: info@whatworks.ed.gov