The What Works Clearinghouse: What You Need to Know SIIA Education Technology DC Fly-In

1. The What Works Clearinghouse: What You Need to Know SIIA Education Technology DC Fly-In April 8, 2003 — Washington, DC —————————————— The What Works Clearinghouse has been established by the U.S. Department of Education’s Institute of Education Sciences to provide educators, policymakers, and the public with a central, independent, and trusted source of scientific evidence of what works in education.

2. Presenters Martin E. Orland Special Assistant, IES Becki Herman WWC Project Director Steve Fleischman WWC Communications Co-leader

3. Educators and Policymakers Need Better Evidence • Demand and a desire for improvement based on what works • Many claims about what works • Few resources to sort through what works • ————————————————— • Result: More than ever, evidence from high quality scientifically based research is needed by education practitioners and policymakers.

4. Help in Sorting Through the Claims • Claims are made by researchers, program and product providers, policymakers, educators, parents, etc. The WWC has been created to help decision-makers. • A new federal initiative (U.S. Department of Education, Institute of Education Sciences) • An ongoing public resource that will assess and report scientific evidence on “What Works?” in education in multiple topic areas • Systematic review processes will be used to report on the quantity, quality, and relevance of evidence, and magnitude of effects of specific educational interventions

5. Who We Are Primary Contractors American Institutes for Research Campbell Collaboration Subcontractors Aspen Systems Caliber Associates Missouri University University of Pennsylvania

6. What Is the WWC? — Principles • Scientific excellence • Independence in scientific decisions • Collaboration • Transparency in processes and standards • Openness to improvement

7. What the WWC is NOT The WWC is not designed to endorse particular educational approaches or interventions. The WWC is not designed to develop and execute field studies on the effects of programs.

8. WWC Activities • Conduct and Publish Reviews of Evidence • Establish standards of excellence for causal inference • Create and work with a respected Technical Advisory Group (TAG) • Select topics for reviews • Produce Evidence Reports • Refine and expand standards • Build awareness and use of WWC products and approaches • Communication and collaboration • Registry of Outcome Evaluators • Online system

9. WWC Technical Advisory Group (TAG) The TAG provides a central and ongoing source of independent, expert input to the WWC. • Composed of experts in social science/educational research methods and measurement • Help establish standards for evidence reviews • Monitor and inform the methodological aspects of the evidence reviews • Review and recommend improvements to the WWC evidence reports • Recommend whether draft reviews of evidence should be entered into the WWC database

10. Larry V. Hedges, Chairperson Betsy Jane Becker Jesse A. Berlin Douglas Carnine Thomas D. Cook David J. Francis Edward H. Haertel Robert L. Linn Mark W. Lipsey David Myers Andrew C. Porter David Rindskopf Cecilia E. Rouse William R. Shadish TAG Members

11. Selection of Topics for Review • Elicit nominations • Website, What Works Network, Public presentations, Education experts • Prioritize • Educational and national importance • Adequate research available • Select and organize • Detailed plan for each year • Long-term plan • Ensure flexibility • Additional reviews

12. Interventions for Beginning Reading Curriculum-based Interventions for Increasing K-12 Math Achievement Preventing High School Dropout Programs for Increasing Adult Literacy Peer Assisted Learning in Elem. Schools: Reading, Mathematics, and Science Gains Interventions to Reduce Delinquent, Disorderly, and Violent Behavior in and out of School Interventions for Elem. English Language Learners: Increasing English Language Acquisition and Academic Achievement Year One Topics

13. Evidence Report Timeline • Production time varies by topic. • Production of Evidence Reports begins as early as April 2003 (4 reports) and as late as November 2003. • Evidence Reports will be released beginning Fall 2003 (3 reports). (see handout for additional details)

14. Steps in Evidence Report (ER) Process & Quality Control

15. WWC and SIIA Members SIIA members are: • in a unique position as producers and consumers of research. • an important source of information and advice to education stakeholders. • involved in a very competitive industry. Therefore, publishers have a strong interest and stake in the work of the WWC and want to ensure that its work is “objective, transparent and follows due process.”

16. SIIA Questions and Concerns • Development Process. In what ways will the WWC provide opportunities to comment on its standards and products? How will it ensure that its standards and processes are “fair, strong and widely supported?” • Validity. What process will the WWC follow in order to ensure that its draft standards and processes are validated prior to their application? How will the WWC avoid subjectivity in its judgments? • Operational Procedures.What is the procedure the WWC will follow to produce Evidence Reports? What will each Evidence Report review? How will topics for review be determined? Will submissions be solicited/accepted?

17. Questions and Concerns (2) • Bias. How will WWC handle potential biases of its evaluators (political, pedagogical, philosophical)? How will the WWC make sure that its selection of topics does not set an education agenda by highlighting only some programs, products, practices or policies? • Scope of Reviewed Research. Is the scope of the WWC’s review too narrow because of its focus on quantitative research designs?

18. Contact Information w-w-c.org • Martin Orland, Special Assistant, IESMartin.Orland@ed.gov 202/219-2297 • Becki Herman, Project Directorrherman@air.org 202/944-5449 • Steve Fleischman, Communications Co-leader sfleischman@eqireports.org 202/296-1902