IES Scientific Standards For Fast Response Projects

1. IES Scientific Standards For Fast Response Projects Presentation to the Regional Educational Laboratory Directors Meeting February 8, 2008

2. Fast Response Projects Provide • Scientifically based information on… • Issues important to regional decision makers

3. NCLB Act Established IES Scientific Standards • Use rigorous, systematic, objective methods • Ensure findings are appropriate to and supported by methods

4. NCLB Act Further Specifies Scientific Research Involves • Systematic empirical methods • Research designs appropriate to question • Reliable measurement, appropriate analyses • Causal claims only from designs supporting them • Replicability • Peer review

5. Guiding Principles of Fast Response Projects 1. Attend to regional needs • Know the audience and purpose 2. Select clear, focused research questions 3. Adhere to IES research standards • Appropriate data sources • Valid, reliable analysis or synthesis • Appropriate, reasonable conclusions

6. An Issues and Answers Report… • Describes a problem, frames a research question • Briefly describes data and analysis methods • Presents findings (and limitations) • Appendices • Allow researchers to assess evidence • Provide supplementary data

7. ATS Peer Review Asks Three Questions 1. Is research question clear, linked to policy issues and knowledge base? 2. Are methods and data appropriate, clearly described? 3. Are findings clearly presented, appropriate?

8. IES Standards:1. Framework and Background

9. Building a Justification for the Research • Report addresses an information need of the region • Interest beyond researchers • Study presents new information • Or makes new use of existing information

10. Use of Prior Literature to Motivate Report Should Be • Balanced • Conflicting evidence • Key publications • Scientific • Discuss strengths and limitations, quality of evidence • Use primary sources

11. When Framing Research Questions • State all questions clearly and in one place • Make them specific • Link questions to the policy issue and regional information need • Describe how answers are useful for the region

12. Use Key Terms for a Broad Readership • Define terms early • Use definitions broadly accepted by researchers . . . • . . . that are clear and understandable for non-researchers • Use terms consistently

13. IES Standards:2. Data Sources and Methods

14. Sources, Data Collection Methods, Sampling • Explain where/who data came from, how you got it, appropriateness for research questions • How consistency achieved across collectors • Describe how you selected sample • Address representativeness; explain outreach • Basics in main body, details in appendix

15. Analysis Methods • Make clear why things were done, not done • Explain how analysis methods are appropriate for research questions • Make clear how findings were derived

16. Specific Analytic Approaches • Quasi-experimental, regression discontinuity studies • Must follow WWC guidelines (evidence screens, standards) • Other quantitative analyses • Cannot address causality • Provide details on multivariate models

17. Specific Analytic Approaches (continued) • Literature reviews • Addressing causality - must follow WWC guidelines (evidence screens, standards) • Not addressing causality - search process, inclusion rules must be clear, systematic

18. Specific Analytic Approaches (continued) • Qualitative analysis (e.g., documents, interview or focus group data) • Explain coding process, inter-rater reliability • Specify unit of analysis • Make clear extent to which participants had chance to (did) respond • Alignment study (curricula, standards, assessments) • Use dimensions that are objective, appropriate to questions

19. Strengths and Limitations • Germane to findings • Address internal validity, generalizability • Don't ignore limits when presenting findings

20. OMB Issues and Confidentiality • Similar data from no more than 9 respondents • No individually identifiable data on students, families, teachers, other staff • No identifiable data on schools, districts that are not publicly available

21. IES Standards:3. Presentation of Findings

22. Answer Research Questions • Align findings with the study’s research questions • Address all the research questions adequately • Make sure readers can find answers easily

23. Support the Findings • Make sure findings clearly supported by the data and analysis • Describe the findings accurately and objectively • Without “un-scientific” adjectives

24. Present Findings Clearly • Organize presentation of findings in a clear, logical fashion • Follow framework supported by literature presented earlier • Make sure reader knows how each finding helps answer a research question

25. Discuss the Implications of the Findings • Show how the findings confirm or disconfirm prior research • Make sure suggestions for further research are feasible, and follow from the findings • Let decision makers decide

26. The Review Process

27. Report Reviews: First Draft ATS Fast Response Lead Liaison between reviewers and IES Reviewer A (ATS Internal)Answers review questions Reviewer B (External Expert) Answers review questions Summary author (ATS Internal) Third reviewer; synthesizes main issues in a summary

28. Report Reviews: Revised Drafts ATS Fast Response Lead Liaison between reviewers and IES Primary Reviewer (1 of 3 first round reviewers) Writes review; areas of assessment primarily driven by first round review Secondary Reviewer (1 of 3 first round reviewers) Adds to, clarifies, questions review of Primary Reviewer