Topics in Special Education Research

1. Topics in Special Education Research Session 3-Experimental & Quasi-experimental Research

2. This Afternoon’s Agenda • Review Course Requirements & Upcoming Assignments • Ethics in Research • Review for, Take, & Correct Quiz • Discussion & Lecture on Experimental Research • Activity • Dismissal

3. Updates/Questions • Discussion Guides- Please submit to dropbox folder and label • 3 main points for all of the readings for that session. • No need to double space • Address all of the headings (even the difficulties, concerns, questions section). • Use complete sentences • These are designed to guide your discussions. • Please put your questions for ME on the feedback guide, so if they aren’t answered by your discussion group, type them on the feedback guide!

4. Upcoming Assignments • Today NIH/CITI training modules • Annotated Bibliography (individual assignment) • Each individual reviews 3 research articles regarding their topic • See Example: http://rxsped596.pbworks.com/w/file/fetch/54804527/Example%20Annotated%20Bibliography.pdf • Conceptual Framework (group assignment) • Group submits short summary of literature and presents a conceptual framework for theories that drive their proposal. • Article Review Assignment (group assignment) • Written Research Proposal (group assignment) • Presentation of Research Proposal (group assignment)

5. Research Question(s) Guidelines • Briefly and clearly state how each research question will be addressed. • For example, “This research question will be answered by comparing the end-of-year state wide test scores of students who received the intervention and those who did not receive the intervention.” • Briefly present the proposed theoretical and practical implications of the findings. • e.g., “The results of this study may have implications for the use of the evidence-based Super-Duper Reading Intervention by elementary special education teachers…”

6. Research Question Tips • Framed based to operationalize (clearly define so that it can be replicated) the objectives of the proposed research project. • Mention the IV and DV and how they will affect each other • Framed based on methodology: • Experimental/Quasi-experimental/Single-subject: “Is there a causal of functional relationships between IV and DV” • Correlational: “Is there a correlational relationship between IV and DV” • Descriptive/Qualitative: Describe a phenomenon or issue better, “What is the prevalence of intellectual disabilities in African-American middle school students?”

7. Ethics in Research • What did you learn from doing the CITI course? • Regarding… • Research misconduct • Data Management • Responsible authorship • Collaboration in Research • http://www.youtube.com/watch?v=W7sfIA1dIGQ • http://www.youtube.com/watch?v=yr5cjyokVUs • http://www.youtube.com/watch?v=b8McGyYAwcU

8. The Belmont Report (1979), National Commission—3 Principles 1. Respect for Persons- • Required to: obtain informed consent & respect privacy of participants 2. Beneficence- • Required to: use best possible design to maximize benefits & minimize harms, show they can perform the procedures and handle the risks, prohibit research that that is without a favorable risk-benefit relationship 3. Justice • Required to: select participants equitably & avoid exploitation of vulnerable populations or populations of convenience.

9. PSU Human Subjects Research Review Committee (HSRC) • http://www.rsp.pdx.edu/compliance_human.php • Portland State University (PSU) is responsible for the rights and welfare of human subjects involved in research sponsored or conducted by the university.   In order to meet this responsibility, the University established the Human Subjects Research Review Committee.  • Members are charged with reviewing all research conducted under the auspices of PSU that involves human subjects to ensure adequate protections are in place.

10. Review for Quiz

11. In-text Citations: Formatting Quotations When quoting, introduce the quotation with a signal phrase. Make sure to include the author’s name, the year of publication, the page number, but keep the citation brief—do not repeat the information. Caruth (1996) states that a traumatic response frequently entails a “delayed, uncontrolled repetitive appearance of hallucinations and other intrusive phenomena” (p.11). A traumatic response frequently entails a “delayed, uncontrolled repetitive appearance of hallucinations and other intrusive phenomena” (Caruth, 1996, p.11).

12. In-text Citations: A Work with Two Authors When citing a work with two authors, use “and” in between authors’ name in the signal phrase yet “&” between their names in parenthesis. According to feminist researchers Raitt and Tate (1997), “It is no longer true to claim that women's responses to the war have been ignored” (p. 2). Some feminists researchers question that “women's responses to the war have been ignored” (Raitt & Tate, 1997, p. 2).

13. In-text Citations: A Work with Three to Five authors When citing a work with three to five authors, identify all authors in the signal phrase or in parenthesis. (Harklau, Siegal, & Losey, 1999) In subsequent citations, only use the first author's last name followed by "et al." in the signal phrase or in parentheses. (Harklau et al., 1999)

14. In-text Citations: A work with 6 or more authors • When citing a work with more than 6 authors, you do NOT have to identify all authors in the signal phrase or parenthesis. • You identify the first author and use et al. after his/her name with the date of the publication (Carr et al., 1999) OR signal phrase in text e.g.: “Carr et al. (1999) conducted a meta-analysis of behavior support practices…”

15. 1. Identify socially important issue 2. Review current literature 3. Define conceptual model 4. Define specific hypothesis(es) and research question(s) 5. Define dependent variable(s)/measure 6. Identify independent variable(s)/measures 7. Select appropriate research design 8. Obtain consents 9. Collect data 10. Analyze data 11. Communicate results Written presentation Oral presentation Steps in the Research/Scientific Process

16. Conceptual Models (or theoretical models) • Theory that drives the research. • Guides our thinking and provides “rules”, “principles” that guides the research and practice. • Structure of assumptions, principles, and rules that holds together the ideas of a broad concept. • Outlines your research

17. “Logical Flaws” of FBA use in public schools (Scott et al., 2005) • FBA is used mainly as a reactionary approach. • opportunity is lost to utilize FBA technology to develop interventions that address minor behaviors that usually precede more serious problems. • FBA is restricted to set of procedures used by “experts” • The rich supply of information from people with whom the student interacts with the most is lost. • FBA is restricted to rigorous procedures that are unrealistic for public school settings. • Disincentive for using FBA technology. • Cynicism as to the practicality of FBA .

18. Proactive…Parsimonious…Practical FBA in schools • FBA conceptualized by Scott et al. as a proactive pre-referral routine that uses the most parsimonious procedures required to create an effective behavior support plan. • Given the time & resource constraints in schools, we must encourage schools to “work smarter” to develop capacity to implement technology to effectively support more students. • Use Practical FBA procedures to develop capacity within a school to utilize FBA technology.

19. Practical FBA Logic Model Individualized Supports 5% of Students Behavior Specialist responsible for 25 FBAs in school of 500 Personnel with “flexible” roles conduct proactive Practical FBA to expand the scope of FBA, prevent intensive problem behaviors, & decrease reliance on specialist. Secondary Group Supports 10-15% of Students School-wide Positive Behavioral Supports 80% of Students

20. Literature Review Guidelines: A) You should educate readers about the topic and provide a clear rationale as to why the study is important and necessary based on the previous research and writing on the topic. B) Within your literature review you will present the logic or conceptual framework as to why and how your current study (topics, methods, designs) is organized the way it is. C) Make this section compelling.Concisely explain the social importance of what you are studying. e.g., Start with a powerful statement or statistic

21. Quiz

22. Correct Quiz

23. Discussion • Get together with 2-3 other people and use your discussion guides to “guide” your discussion. • Spend more time on the chapter reading and the Quality Indicator Article as we will be doing an activity on the other two articles later!

24. Lecture • Experimental & Quasi Experimental Research • Research Designs & Threats to Validity • Attempt tonight to apply the principle of teaching “less more thoroughly”…. • https://www.youtube.com/watch?v=qtLnBz6lbRQ

25. Experimental Design • “Gold Standard” in Research (Moore & McCabe, 1993; Feuer, Towne, & Shavelson, 2002; Slavin, 2002). • Rooted in postpositivist paradigm. • Seeks to make causal conclusions. • Difference between experimental design and quasi-experimental design is the use of random selection of participants.

26. In both experimental & quasi-experimental designs…. • Emphasis is on operationally defining the variables (dependent & independent) and the context of the research. • Dependent variable(s)- outcome variables (e.g., reading scores) • Independent variable(s)- variable that is manipulated (the intervention or practice; e.g., reading curriculum) • Context- defined clearly so replication can occur (e.g., K-3 school with 200 students, etc.)

27. Experimental Group vs Control Group • Experimental (or treatment group)- receives the intervention • Control group- business as usual. • For true “experimental” research, participants are randomly assigned to each group. • In order for to be considered random, every person must have an equal chance of being in either group

28. Experimental and quasi-experimental research involves… • Direct manipulation of an independent variable (intervention)

29. Validity • Refers to whether a study is able to scientifically answer the questions it is intended to answer. • Extent to which your test (or study) measures what it intends to measure.

30. Internal Validity • Changes observed in the dependent variable (outcome) are due to the effect of the independent variable (intervention)…..& not to some other unintended variables (extraneous, alternative explanations) • 12 threats to internal validity (noted by Mertens, 2010) • https://www.youtube.com/watch?v=_UPUtlHDM0A

31. 12 Threats to Internal Validity (think when something other than the IV affect the results of a study) • History- events other than IV affected results? • Maturation- changes in participants? • Testing- participants became “test-wise”? • Instrumentation- difference between pre- and posttests? • Statistical regression- extreme groups used? • Differential selection- groups differed in ways other than exposure to IV?

32. Threats to Internal Validity continued 7. Experimental mortality- drop outs of study? 8. Selection-maturation- was selection a problem based on the characteristics of the sample? (e.g., participants in one group may have been older) 9. Experimental treatment diffusion- treatment & control groups shared information? 10. Compensatory rivalry by the control group 11. Compensatory equalization of treatments –extra resources given to control group? 12. Resentful demoralization of the control group

33. External Validity (think generalizability) • External Validity= extent to which findings in one study can be applied to another situation. • AKA: ecological validity, generalizability • 10 threats posed as questions (noted by Mertens, 2010)

34. 10 Threats (questions) to External Validity • Were the variables, context, and treatment described in sufficient detail? • Were multiple treatments used? Did they interfere with each other? • Was the Hawthorne effect (special attention as part of study affecting results) operating? • Was the treatment influenced by being novel or disruptive? • What was the influence of the individual experimenter?

35. Questions to external validity continued 6. Were the participants sensitized by taking a pretest? 7. Were the participants sensitized by taking a posttest? 8. What was the influence of the type of measurement used for the dependent variable? 9. Was there an interaction of history and treatment effects? 10. What was the influence of the time period that elapsed between the treatment and administration of the dependent variable?

36. Treatment Fidelity…another threat • Treatment fidelity- implementer of the independent variable follows the exact procedures specified for administering the treatment(s).

37. Strength of treatment…another threat • May not be reasonable to expect participants to improve given the “strength” of the intervention. • Intervention may not have been tried long enough or delivered with adequate intensity.

38. Coding system used for research designs • R= Random assignment of subjects to conditions • X=Experimental treatment (e.g., reading curriculum) • O= Observation of the dependent variable (e.g., test or observation measure)

39. Single Group Designs • One-shot case study X O Threats: History, maturation, & mortality (drop out) Other concerns using this design? No control group & No pretest to know if it was the intervention that affected outcome. Very weak design

40. One-Group Pretest-Posttest Design O X O Threats: • History, maturation, • What would help control for these threats? • Control group- both groups taking the tests at same time, but one not receiving the intervention • But sometimes it is difficult to find a control group

41. Time Series Design • Involves measurement of the dependent variable at periodic intervals. O OOOO X O OOOO • If behavior is stable in baseline (before intervention), then change can be attributed to intervention. • Controls for several threats • Maturation, testing, differential selection (same persons involved) • https://www.youtube.com/watch?v=GUq_tO2BjaU

42. Experimental Designs • Pre-test-Post-test Control Group Design R O X O R O O Controls for what threats? history, differential selection, mortality (pre-test can show differences in drop-outs) Posttest-only Control Group Design R X O R O • Controls for what threats? • Same as above, except for mortality (no pretest)

43. Single-Factor Multiple-Treatment Designs X1= intervention 1 X2= intervention 2 ……. R O X1 O R O X2 O R O O Controls for threats because participants randomly assigned to comparison groups and pre-post-tests conducted.

44. Solomon 4-group Design • If worried about pretesting affecting validity R O X O R O O R X O R O https://www.youtube.com/watch?v=7RRVW4iO7gA

45. Quasi-experimental designs • Similar designs to experimental designs, except for…… • Lacking…. • Random assignment • Can’t make a strong causal statement • https://www.youtube.com/watch?v=_WpBxERGNVw

46. How do we know if a research study involves rigorous, systematic and objective procedures? • CEC-Division for Research • Sponsored prominent researchers to author papers to propose • Parameters for establishing that reported research has been conducted with high quality (quality indicators) • Criteria for determining whether a practice has been studied sufficiently (enough high-quality research studies conducted on its effectiveness) and shown to improve student outcomes (effects are strong enough) Graham, S. (2005). Criteria for evidence-based practice in special education [special issue]. Exceptional Children, 71.

47. Exceptional Children(2005) volume 71(2) • Group Experimental and Quasi-Experimental Research(Gersten, Fuchs, Comptom, Coyne, Greenwood, & Innocenti) • Single-Subject Research(Horner, Carr, Halle, McGee, Odom, & Wolery) • Correlational Research(Thompson, Diamond, McWilliam, Snyder, Snyder) • Qualitative Studies (Brantlinger, Jimenez, Klingner, Pugach, & Richardson)

48. Quality Indicators (QIs) for Experimental (and Quasi-Experimental) Research • Describing Participants • Sufficient information about participants and interventionists, selection procedures as well as comparability across conditions • Implementation of Intervention and Description of Comparison Conditions • Clear description of intervention (and comparison conditions) with implementation fidelity assessed • Outcome Measures • Use of multiple measures at appropriate times • Data Analysis • Analysis techniques appropriate to questions and unit of analysis with effect size calculated

49. Sampling Strategies • Probability-Based • Non-probability Based • Examples???

50. Random vs Non-random sampling