Experimental RESEARCH DESIGNS

1. ExperimentalRESEARCHDESIGNS This presentation was prepared by Del Siegle. Some of the material is from an earlier presentation by Scott Brown.

2. Experimental Research Designs have Two Purposes: • …to provide answers to research questions • ...to control variance (differences)

3. The main function of the experimental research design is to control variance. • Principle: maximize systematic variance, control extraneous systematic variance, and minimize error variance. In other words control variance.

4. Therefore the researcher attempts to: • maximize the variance of the variable(s) of the research hypothesis (i.e., maximize the difference in the dependent variable [outcome] caused by maximizing the differences in the independent variable [treatment]). • control the variance of extraneous or "unwanted" variables that may have an effect on the experimental outcomes, but which he/she is not interested (limit factors other than the treatment (IV) that could be causing differences in the outcome (DV) . • minimize the error or random variance (i.e., avoid unreliable measurement instruments which have high errors of measurement ).

5. Maximization of Experimental Variance • experimental variance • the variance due to the manipulated (i.e., treatment) or attribute (i.e., gender) variables (IV) research precept: • design, plan and conduct research so that experimental conditions are as different as possible on the independent variable.

6. Control of Extraneous Variables (EV) • eliminate the variable (i.e., if you are worried about gender, only include one gender in the study). • randomization (i.e., if you randomly assign subjects to groups, the extraneous variable should be equally distributed among the groups) • build it into the design – make it a moderator variable (i.e., if you are worried about gender, build it into the analysis [2-way ANOVA]—we’ll learn about this later) • match subjects (i.e., match the characteristics of subjects and put one of each matched pair in each group) • statistically equate groups (i.e., use ANCOVA [Analysis of Covariance] to analyze the data with the extraneous variable used as a covariate—we’ll learn about this later)

7. Minimizing Error Variance has Two Principle Aspects: • reduction of errors of measurement through controlled conditions (i.e., standardize testing procedures) • increase in the reliability of measures (i.e., revise test instruments or find more reliable ones)

8. Experimental Designs Should be Developed to Ensure Internal and External Validity of the Study

9. Internal Validity: • Are the results of the study (DV) caused by the factors included in the study (IV) or are they caused by other factors (EV) which were not part of the study? There are 16 common threats to internal validity.

10. Threats to Internal Validity Subject Characteristics (Selection Bias/Differential Selection) -- The groups may have been different from the start. If you were testing instructional strategies to improve reading and one group enjoyed reading more than the other group, they may improve more in their reading because they enjoy it, rather than the instructional strategy you used.

11. Threats to Internal Validity Loss of Subjects (Mortality) -- All of the high or low scoring subject may have dropped out or were missing from one of the groups. If we collected posttest data on a day when the honor society was on field trip at the treatment school, the mean for the treatment group would probably be much lower than it really should have been.

12. Threats to Internal Validity Location Perhaps one group was at a disadvantage because of their location.  The city may have been demolishing a building next to one of the schools in our study and there are constant distractions which interfere with our treatment.

13. Threats to Internal Validity The testing instruments may not be scores similarly. Perhaps the person grading the posttest is fatigued and pays less attention to the last set of papers reviewed. It may be that those papers are from one of our groups and will received different scores than the earlier group's papers Instrumentation Instrument Decay

14. Threats to Internal Validity The subjects of one group may react differently to the data collector than the other group. A male interviewing males and females about their attitudes toward a type of math instruction may not receive the same responses from females as a female interviewing females would. Data Collector Characteristics

15. Threats to Internal Validity The person collecting data my favors one group, or some characteristic some subject possess, over another. A principal who favors strict classroom management may rate students' attention under different teaching conditions with a bias toward one of the teaching conditions. Data Collector Bias

16. Threats to Internal Validity Testing The act of taking a pretest or posttest may influence the results of the experiment. Suppose we were conducting a unit to increase student sensitivity to prejudice. As a pretest we have the control and treatment groups watch Shindler's List and write a reaction essay. The pretest may have actually increased both groups' sensitivity and we find that our treatment groups didn't score any higher on a posttest given later than the control group did. If we hadn't given the pretest, we might have seen differences in the groups at the end of the study.

17. Threats to Internal Validity History Something may happen at one site during our study that influences the results. Perhaps a classmate dies in a car accident at the control site for a study teaching children bike safety. The control group may actually demonstrate more concern about bike safety than the treatment group.

18. Threats to Internal Validity There may be natural changes in the subjects that can account for the changes found in a study. A critical thinking unit may appear more effective if it taught during a time when children are developing abstract reasoning. Maturation

19. Threats to Internal Validity Hawthorne Effect The subjects may respond differently just because they are being studied. The name comes from a classic study in which researchers were studying the effect of lighting on worker productivity. As the intensity of the factory lights increased, so did the worker productivity. One researcher suggested that they reverse the treatment and lower the lights. The productivity of the workers continued to increase. It appears that being observed by the researchers was increasing productivity, not the intensity of the lights.

20. Threats to Internal Validity One group may view that it is in competition with the other group and may work harder than they would under normal circumstances. This generally is applied to the control group "taking on" the treatment group. The terms refers to the classic story of John Henry laying railroad track. John Henry Effect

21. Threats to Internal Validity The control group may become discouraged because it is not receiving the special attention that is given to the treatment group. They may perform lower than usual because of this. Resentful Demoralization of the Control Group

22. Threats to Internal Validity Regression (Statistical Regression) -- A class that scores particularly low can be expected to score slightly higher just by chance. Likewise, a class that scores particularly high, will have a tendency to score slightly lower by chance. The change in these scores may have nothing to do with the treatment.

23. Threats to Internal Validity The treatment may not be implemented as intended. A study where teachers are asked to use student modeling techniques may not show positive results, not because modeling techniques don't work, but because the teacher didn't implement them or didn't implement them as they were designed. Implementation

24. Threats to Internal Validity Someone may feel sorry for the control group because they are not receiving much attention and give them special treatment. For example, a researcher could be studying the effect of laptop computers on students' attitudes toward math. The teacher feels sorry for the class that doesn't have computers and sponsors a popcorn party during math class. The control group begins to develop a more positive attitude about mathematics. Compensatory Equalization of Treatment

25. Threats to Internal Validity Experimental Treatment Diffusion Sometimes the control group actually implements the treatment. If two different techniques are being tested in two different third grades in the same building, the teachers may share what they are doing. Unconsciously, the control may use of the techniques she or he learned from the treatment teacher.

26. Once the researchers are confident that the outcome (dependent variable) of the experiment they are designing is the result of their treatment (independent variable) [internal validity], they determine for which people or situations the results of their study apply [external validity].

27. External Validity: • Are the results of the study generalizable to other populations and settings? External validity comes in two forms: population and ecological.

28. Threats to External Validity (Population) Population Validity is the extent to which the results of a study can be generalized from the specific sample that was studied to a larger group of subjects. It involves... • ...the extent to which one can generalize from the study sample to a defined population--If the sample is drawn from an accessible population, rather than the target population, generalizing the research results from the accessible population to the target population is risky. • ...the extent to which personological variables interact with treatment effects--If the study is an experiment, it may be possible that different results might be found with students at different grades (a personological variable).

29. Threats to External Validity (Ecological) Ecological Validity is the extent to which the results of an experiment can be generalized from the set of environmental conditions created by the researcher to other environmental conditions (settings and conditions). There are 10 common threats to external validity.

30. Explicit description of the experimental treatment Threats to External Validity (Ecological) (not sufficiently described for others to replicate) If the researcher fails to adequately describe how he or she conducted a study, it is difficult to determine whether the results are applicable to other settings.

31. Threats to External Validity (Ecological) Multiple-treatment interference (catalyst effect)If a researcher were to apply several treatments, it is difficult to determine how well each of the treatments would work individually. It might be that only the combination of the treatments is effective.

32. Threats to External Validity (Ecological) Hawthorne effect (attention causes differences)Subjects perform differently because they know they are being studied. "...External validity of the experiment is jeopardized because the findings might not generalize to a situation in which researchers or others who were involved in the research are not present" (Gall, Borg, & Gall, 1996, p. 475)

33. Threats to External Validity (Ecological) Novelty and disruption effect (anything different makes a difference)A treatment may work because it is novel and the subjects respond to the uniqueness, rather than the actual treatment. The opposite may also occur, the treatment may not work because it is unique, but given time for the subjects to adjust to it, it might have worked.

34. Threats to External Validity (Ecological) • (it only works with this experimenter)The treatment might have worked because of the person implementing it. Given a different person, the treatment might not work at all. Experimenter effect

35. Threats to External Validity (Ecological) Pretest sensitization (pretest sets the stage)A treatment might only work if a pretest is given. Because they have taken a pretest, the subjects may be more sensitive to the treatment. Had they not taken a pretest, the treatment would not have worked.

36. Threats to External Validity (Ecological) Posttest sensitization (posttest helps treatment "fall into place")The posttest can become a learning experience. "For example, the posttest might cause certain ideas presented during the treatment to 'fall into place' " (p. 477). If the subjects had not taken a posttest, the treatment would not have worked.

37. Threats to External Validity (Ecological) Interaction of history and treatment effect (...to everything there is a time...)Not only should researchers be cautious about generalizing to other population, caution should be taken to generalize to a different time period. As time passes, the conditions under which treatments work change.

38. Threats to External Validity (Ecological) Measurement of the dependent variable (maybe only works with M/C tests)A treatment may only be evident with certain types of measurements. A teaching method may produce superior results when its effectiveness is tested with an essay test, but show no differences when the effectiveness is measured with a multiple choice test.

39. Threats to External Validity (Ecological) Interaction of time of measurement and treatment effect (it takes a while for the treatment to kick in)It may be that the treatment effect does not occur until several weeks after the end of the treatment. In this situation, a posttest at the end of the treatment would show no impact, but a posttest a month later might show an impact.

40. First, and foremost, an experiment must have internal validity. If the researchers cannot be certain that the results of the experiment are dependent on the treatment, it does not matter to which people or situations they wish to generalize (apply) their findings. The importance of external validity is reliant on having internal validity in much the same way that the validity of a measurement instrument is reliant on the instrument being reliable. However, the more tightly experimenters design their study, the more they limit the populations and settings to whom they can generalize their findings.

41. The next section will describe different research designs.

42. Suppose a researcher wants to study the effect of a reading program on reading achievement. She might implement the reading program with a group of students at the beginning of the school year and measure their achievement at the end of the year. X O This simple design is known as a one-shot case study design.

43. Unfortunately, the students’ end of year reading scores could be influenced by other instruction in school, the students’ maturation, or the treatment. We also do not know whether the students’ reading skills actually changed from the start to end of the school year. We could improve on this design by giving a pretest at the start of the study. O X O This is known as a one-group pretest-posttest design.

44. O O Unfortunately, the students’ end of year reading scores still could be influenced by other instruction in school, the students’ maturation, or the treatment. Our researcher may wish to have a comparison group. O X O This is a static-group pretest-posttest design.

45. If our researcher believes that the pretest has an impact on the results of the study, she might not include it. O X O O O This is a static-group comparison design.

46. Because our researcher did not pretest, she might wish to randomly assign subjects to treatment and control group. Random assignment of subject to groups should spread the variety of extraneous characteristics that subjects possess equally across both groups. R O X O R O O This is a randomized posttest-only, control group design.

47. O O Of course, our researcher could also include a pretest with her random assignment. R X O R O This is a randomized pretest-posttest control group design.

48. R O X O R O O R X O R O Occasionally researchers combine the randomized pretest-posttest control group design with the randomized posttest-only, control group design. This is a randomized Solomon four-group design.

49. R O O R O O R O R O and given the posttest. With the randomized Solomon four-group design, all groups are randomly assigned Two of the groups are given pretests. X One of the pretest groups is assigned to treatment and one of the non-pretest groups is assigned to treatment. X

50. Each of the designs described in this section has advantages and disadvantages that influence the studies internal and external validity. This presentation was prepared by Del Siegle. Some of the material is from an earlier presentation by Scott Brown.