Chapter 10: Experimental Research

1. Chapter 10: Experimental Research • Objectives: • Briefly state the purpose of experimental research and list the basic steps involved in conducting and controlling an experiment. • Briefly define internal validity and describe eight major threats to the internal validity of an experiment.

2. Chapter 10: Experimental Research • Objectives: • Briefly define external validity and describe six major threats to the external validity of an experiment. • Identify and briefly describe five ways to control extraneous variables.

3. Chapter 10: Experimental Research • Objectives: • Define and provide examples of single-variable designs (i.e., pre-experimental, true experimental, and quasi-experimental designs) and factorial designs, and explain how they differ.

4. Experimental Research • Experimental research is the only type of research that can test hypotheses to establish cause-effect relations. • The researcher manipulates at least one independent variable and controls other relevant variables, and observes the effect on one or more dependent variables. • The researcher manipulates the treatment. • The researcher has control over selection and assignment.

5. Experimental Research • In experimental research studies the independent variable is also called the treatment, causal, or experimental variable. • In experimental research studies the dependent variable is also called the criterion, effect, or posttest variable.

6. Experimental Research • Experimental research is the most structured of all research. • When conducted well, experimental research studies can provide evidence for cause-effect relations. • Several experimental studies taken together can provide support for generalization of results.

7. Experimental Research • The experimental process • The steps in the experimental research process are the same as in other types of research. • Selecting and defining a problem • Selecting participants and measuring instruments • Preparing a research plan • Executing procedures • Analyzing the data • Formulating conclusions

8. Experimental Research • The experimental process • In experimental studies, the researcher controls selection and assignment. • Experimental studies often examine comparisons between or among groups. • Comparison of approaches (A versus B) • Comparison of an approach to an existing approach (A versus no A) • Comparison of different amounts of a single approach (A little of A versus a lot of A)

9. Experimental Research • The experimental process • In experimental research studies the group that receives the treatment is the experimental group and the group that does not receive the treatment is called the control group. • Sometimes groups are comparison groups that receive alternative treatments (e.g., two types of instruction in a content area).

10. Experimental Research • Manipulation and control • In experimental studies, researchers control or remove the influence of extraneous variables. • Participant variables • Organismic (e.g., age) • Intervening (e.g., interest) • Environmental variables (e.g., school or teacher effects)

11. Threats to Experimental Validity • Internal validity refers to the degree to which observed differences in the dependent variable are a direct result of manipulation of the independent variable and not some other variable. • Internal validity is concerned with rival explanations for an effect.

12. Threats to Experimental Validity • External validity, sometimes referred to as ecological validity, is the degree to which the results from a study are generalizable to other groups. • When researchers increase the internal validity of their study, they decrease their external validity.

13. Threats to Experimental Validity • When researchers are concerned with external validity, their ability to control important extraneous variables suffers. • When there is a choice, researchers should err on the side of control and maximize internal validity.

14. Threats to Internal Validity • History • Maturation • Testing • Instrumentation • Statistical regression • Differential selection of participants • Mortality • Selection-maturation and interactive effects

15. Threats to Internal Validity • History • Any event occurring during a study that is not part of the experimental treatment but that may effect the dependent variable represents a history threat. • Longer-lasting studies are more prone to history threats.

16. Threats to Internal Validity • History threat example • In a study of the effects of instructional simulations in learning chemistry content, a history threat would be demonstrated if students in the study were exposed to simulations in a different setting, such as when learning geography, while the study was being conducted.

17. Threats to Internal Validity • Maturation • Maturation refers to physical, intellectual, and emotional changes that naturally occur within participants over a period of time.

18. Threats to Internal Validity • Maturation threat example • In studies of interventions that are designed to increase children’s theory of mind, if the interventions lasted more than a couple of weeks at critical time points, participants may gain critical theory of mind awareness simply due to cognitive development and not due to the treatment.

19. Threats to Internal Validity • Testing • Testing as a threat to internal validity is demonstrated when taking a pretest alters the result of a posttest.

20. Threats to Internal Validity • Instrumentation • Instrumentation is a threat to internal validity when the instrumentation is either unreliable or is changed between pre- and posttesting.

21. Threats to Internal Validity • Statistical regression • Extremely high or low scores tend to regress to the mean on retesting. • Statistical regression example • If students perform poorly on a pretest it is difficult to determine if the gain in their scores is due to treatment effects.

22. Threats to Internal Validity • Differential selection of participants • Participants in the control and experimental groups differ in ways that influence the dependent measure.

23. Threats to Internal Validity • Mortality • Participants drop out of the study at differential rates across conditions.

24. Threats to Internal Validity • Selection can interact with other threats to internal validity (i.e., history, maturation, instrumentation). • Participants selected into the treatment and control conditions have different experiences or maturation rates or instrumentation varies across conditions.

25. Threats to External Validity • External validity threats can be divided into two categories: • ‘Generalizing to whom’ threats • Threats affecting groups to which the study can be generalized • ‘Generalizing to what’ threats • Threats affecting the settings, conditions, variables, and contexts to which the results can be generalized

26. Threats to External Validity • Pretest-treatment interaction • Multiple-treatment interference • Selection-treatment interaction • Specificity of variables • Experimenter effects • Reactive arrangements

27. Threats to External Validity • Reactive arrangements • Hawthorne effect: Any situation in which participants’ behavior is affected because they are in a study. • John Henry effect (Compensatory rivalry): Members of the control group compete with the experimental group.

28. Threats to External Validity • Reactive arrangements • Placebo effect: To combat compensatory rivalry, researchers attempt to give control groups a placebo, not the experimental treatment, but something to decrease the perception that they are in the control group. Participants should perceive they are all getting the same thing. • Novelty effect: When participants are engaged in something different this may increase attention, interest, behavior, learning, etc., just because it is something new.

29. Group Experimental Designs • The validity of an experiment is a function of the degree to which extraneous variables are controlled. • Randomization is the best mechanism to control for extraneous variables. • Randomization distinguishes experimental designs. • Randomization should be used whenever possible. • If groups cannot be randomly formed, variables should be held constant when at all possible (e.g., time of day, which researcher is present).

30. Group Experimental Designs • Participant variables can be controlled and held constant • Matching can equate groups through random assignment of pairs. • Comparing homogeneous groups allows the researcher to control for extraneous variables.

31. Types of Group Designs • Single-variable designs are any design that involves one manipulated variable. • Pre-experimental designs do not adequately control for extraneous variables and should be avoided. • True-experimental designs offer a very high degree of control and are always preferred designs. • Quasi-experimental designs do not control as well as experimental designs but are preferable over pre-experimental designs.

32. Pre-Experimental Designs • The one-shot case study involves a single group that is exposed to a treatment (X) and then posttested (O). X O • Threats to validity are not adequately controlled with this design. • Do not use this design.

33. Pre-Experimental Designs • The one-group pretest-posttest design involves a single group that is pretested, exposed to treatment, and then tested again. O X O • The success of the treatment is determined by comparing pretest and posttest scores. • This design does not control for history, testing, instrumentation, regression, or maturation. • Statistical regression is not controlled nor is pretest-treatment interaction.

34. Pre-Experimental Designs • The static-group comparison design involves at least two nonrandomly formed groups. One group receives an experimental treatment and the other group receives the traditional treatment. Both groups are posttested. X1 O X2 O

35. Pre-Experimental Designs • The number of groups can be expanded beyond two. • The groups are better described as comparison, not experimental and control. • This design does not control for maturation, selection effects, selection interactions, and mortality.

36. Pre-Experimental Designs • There is some control for history in this design. • This design is sometimes used in exploratory studies.

37. True Experimental Designs • The pretest-posttest control group design requires at least two groups. • Groups are formed by random assignment. • Both groups are administered a pretest, each group receives a different treatment and both groups are posttested. • The design may be extended to include additional groups.

38. True Experimental Designs R O X1 O R O X2 O R O X3 O • The combination of random assignment and the presence of a pretest and a control group serve to control for all threats to internal validity.

39. True Experimental Designs • The only potential weakness in this design is a possible interaction between the pretest and the treatment. • Researchers should report assess and report the probability of a pretest-treatment interaction.

40. True Experimental Designs • There are a few variations on the basic pretest-posttest control group design. • One variation includes random assignment of matched pairs to the treatment groups. • There is little advantage to this variation. • Another variation of this design involves one or more additional posttests. R O X1 O O R O X2 O O

41. True Experimental Designs • The posttest-only control group design is the same as the pretest-posttest control group design except that it lacks a pretest. • This design is often expanded to include more than two groups. R X1 O R X2 O

42. True Experimental Designs • The posttest-only control group design is best used when there is likelihood of a pretest-treatment interaction threat. • As with the pretest-posttest control group design, the addition of a matched random assignment does not represent an increased advantage.

43. True Experimental Designs • The Solomon Four-Group Design is a combination of the pretest-posttest control group design and the posttest-only control group design. R O X1 O R O X2 O R X1 O R X2 O

44. True Experimental Designs • The analysis of the Solomon four-group design is a 2 x 2 factorial analysis of variance. • This analysis tests if those who received the treatment performed differently than those who did not. • This analysis can assess if there is a testing effect. • This analysis assesses for pretest-interaction effects.

45. True Experimental Designs • The Solomon four-group design requires a large number of participants. • The Solomon four-group design may not always be the best design. • The design selected should be based upon potential threats and the nature of the proposed study.

46. Quasi-Experimental Designs • When it is not possible to assign participants to groups randomly, researchers can use quasi-experimental studies. • In the nonequivalent control group design, two or more treatment groups are pretested, administered a treatment, and posttested. O X1 O O X2 O

47. Quasi-Experimental Designs • The nonequivalent control group designinvolves the random assignment of groups not individuals. • The lack of random assignment introduces validity threats (e.g., regression, and selection interaction effects). • To reduce threats when using this design researchers often assure groups are as equivalent as possible (e.g., use ANCOVA).