Don’t Be Fooled: Assessing the Quality of Evidence

1. Don’t Be Fooled: Assessing the Quality of Evidence District Review and Selection Process Kim St. Martin (MIBLSI) & Brad Niebling (IA Dept of Education 10/12/17

2. Outcomes • Considerations for judging the quality of evidence • Describe benefits of using an effective innovation review and selection process that includes a review of evidence and other critical factors • Identify the components of an effective innovation review and selection process • Review an example • Demonstrate how the review and selection process can also be used for “de-selection”

3. 1.0 Quality “Evidence:” Good, Bad and the Ugly

4. Why Evidence-Based Practice? • Increase probability of success for more students • Central feature in implementation of MTSS & RtI • Required in Every Student Succeeds Act (ESSA); many states require it as well

5. Levels of Evidence Pyramid

6. Major Categories • Filtered information: judges the quality of studies and makes recommendations for practice • Includes the following: • Systematic reviews • Critically-appraised topics • Critically-appraised individual articles

7. Major Categories (cont.) • Unfiltered Information: includes the primary, unfiltered literature that requires people accessing to take on the role of making sure the information you found is reliable and valid • Includes the following: • Randomized controlled trials • Cohort studies • Case-controlled studies; case series / reports

8. 1. Filtered Information • Systematic Review: • Ask a specific question • Thorough literature review across studies • Eliminate poorly done studies • Attempt to make practice recommendations based on well-done studies • Critically -Appraised Topics: • Evaluate and synthesize • Multiple research studies • Resulting in short, systematic reviews about a particular topic • Critically-Appraised individual articles • Evaluate individual research studies • Write a short synopsis of the study

9. 2. Unfiltered Information • Randomized Control Trials: • Random assignment • Intervention group (thing studying to determine its effect) • Control group • Considered to be superior to other study designs • Cohort Studies: • Observes a group of individuals • Meet a set of criteria (selection criteria) • Longitudinal (cohort is followed over time) • Case-Controlled Studies; Case Series / Reports: • Observational study • Participants are not randomly assigned to the intervention or control group • Lowest level of rigor for research designs • Results may be confounded by other variables

10. Other Research Design Types • Quasi-experimental: looks like an experimental design but lacks the key ingredient of “random assignment” • Concerns about internal validity because the intervention group and control groups may not be comparable at the onset of the study (baseline) • Institute of Education Sciences (IES) has provided guidance on designing quasi-experimental studies (https://ies.ed.gov/ncee/wwc/Multimedia/23)

11. Other Research Design Types • Regression discontinuity (RDD): assignment to intervention or comparison group based on performance relative to cutoff value (e.g, score on a test) • This is a regression analysis • Concerns about integrity of forcing variable, attrition, strict parameters for modeling and analyses • Institute of Education Sciences (IES) has provided guidance on designing and evaluating RDD studies (https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_standards_handbook_v4_draft.pdf)

12. Activity 1.0 • Review the levels of evidence presented in this section making note of critical pieces of information you believe your colleagues would benefit from hearing about after this session • Your presenters will put the group into a partner activity after reviewing the slides

13. Activity 1.1 • Describe the elements of your district’s process for conducting a thorough review of evidence prior to selection • If you are unsure about your district’s process, identify some information you want to know after leaving this conference • Discuss your district’s capacity (people, knowledge/skills, budget) to review evidence programs and practices

14. Evidence is Only Part of the Equation Reviewing the evidence of programs, practices, etc. is only part of the equation. Districts need to consider other critical factors to help them make good decisions about what to select for improving student outcomes.

15. 2.0 Benefits of a Review and Selection Process

16. Key Terminology • Effective Innovation: A set of defined practices used in schools to achieve outcomes that have been empirically proven to produce desired results • To be an effective innovation, the practices, programs, assessments, or initiatives should be proven to be “usable:” teachable, learnable, doable, and readily assessed in practice

17. Benefits • Increased confidence in the following: • Participation in initiatives or the adoption of programs, practices, and assessments are the best available • District has a full understanding of the resources needed to successfully use the selected effective innovations • Decisions that resulted in not selecting an effective innovation or de-selecting the use of an existing innovation resulted in a thorough analysis and critical factors

18. 3.0 Effective Innovation Review and Selection Process

19. Components of a Review and Selection Process • Purpose of a review and selection process • Guidelines for when to use the process • Decision-making protocol • Directions for: • Completing the review and selection tool • Providing supporting documentation for specific items • Submission

20. 1. Purpose • Include the following: • Brief summary of the purpose and intended outcome of conducting a thorough review of an effective innovation • Rationale about why the district expects a thorough review process be completed before decisions are made to select an effective innovation

21. 2. Guidelines for Use • List likely scenarios that would warrant the use of the review process: • Approached to consider participation in an initiative, “pilot project,” and / or approached to use a new assessment or data system • Considering purchasing new curriculum resource materials • Considering purchasing new assessments, data systems, or educational software • Considering to continue the use of effective innovations that overlap or appear to be redundant with other effective innovations (de-selection)

22. 3. Decision-Making Protocol • List the people with the highest level of decision-making authority to determine whether the review process will result in a new selection or de-selection • Include statements about the conditions that would warrant involvement from other groups / teams (e.g., board of education, curriculum council) • Provide parameters for timelines to make decisions

23. 4. Directions • Steps outline: • Initiating a review / selection process (e.g., who can do this and what needs to occur before it is started) • People that need to be involved in the process: • Specific items in the review and selection tools are to be completed by pre-determined designees • Parameters for seeking consultation from program, assessment developers, or certified individuals to adequately represent the effective innovation • Submitting the tool with the appropriate documentation

24. Selection Tools • Two tools: • Program, Practice, or Initiative review tool • Assessment review tool • Each tool is framed around six critical factors that need to be considered during a high-quality review and selection process

25. Six Critical Factors • Effective innovation overview (e.g., name / title, description) • Need for the innovation • Fit with district, state, national priorities or existing efforts • Resources necessary to support peoples’ successful use • Evidence to demonstrate positive effect on outcomes • Readiness for the effective innovation’s use in a typical school / classroom setting • Is the effective innovation mature enough to use in a typical classroom setting and do staff meet the qualifications to successfully use the effective innovation?)

26. Hexagon Tool (Blasé, K., Kiser, L., Van Dyke, M., 2013)

27. Activity 3.0 • Given the components of a review and selection process (slide 16) and the six critical factors (slide 22), outline areas you believe your district’s process overlaps and opportunities to strengthen your existing process

28. 4.0 Review and Selection Template

29. Activity 4.0 • Independently read page 1 of the document, “Effective Innovation Review and Selection Process” • Jigsaw Activity: • Partner 1: Read the directions and steps for the Program, Practice, or Initiative Review process (pp 2-7) • Partner 2: Read the directions and steps for the Assessment Review and Selection Tool (pp. 8-14) • Each partner will develop talking points outlining the important sections, reasons for their importance, and critical areas that are included in the tool that could get overlooked

30. 5.0 Additional Resources and Examples

31. Additional Resources • Core Reading Curriculum Analysis Process: used prior to the district completes the Effective Innovation Review and Selection Tool (handout) • Example of a district’s Effective Innovation Review and Selection Process (electronic access) • Example of how a district engaged in a de-selection process to make room for the new effective innovation (electronic access) • List of information on evidence-based programs and practices (handout)

32. Activity 5.0 • With your partner review the additional resources • Given the context of your district, what information do you want to share with your colleagues

33. Thank You! Brad Niebling, Ph.D. Iowa Dept. of Education, Bureau Chief brad.niebling@iowa.gov Kim St. Martin, Ph.D. MIBLSI Assistant Director kstmartin@miblsimtss.org