Educational Research: Experimental Studies

Educational Research: Experimental Studies EDU 8603 Educational Research Richard M. Jacobs, OSA, Ph.D.

Research... • The systematic application of a family of methods employed to provide trustworthy information about problems …an ongoing process based on many accumulated understandings and explanations that, when taken together lead to generalizations about problems and the development of theories

The basic steps of research... Scientific and disciplined inquiry is an orderly process, involving: recognition and identification of a topic to be studied (“problem”) description and execution of procedures to collection information (“method”) objective data analysis statement of findings (“results”)

Research methods... Quantitative… …collects and analyzes numerical data obtained from formal instruments

Quantitative methods... • descriptive research (“survey research”) • correlational research • causal-comparative research (“ex post facto research”) • experimental research

experimental research …the researcher selects participants and divides them into two or more groups having similar characteristics and, then, applies the treatment(s) to the groups and measures the effects upon the groups

Conducting an experimental study… 1. select the problem 2. select participants and instrument 3. selection and execution of a research plan 4. data analysis and formulation of conclusions

Types of experimental comparison… 1. comparison of two different approaches (A versus B) 2. comparison of an existing approach to a new approach (A and ~ A) 3. comparison of differing amounts of a single approach (A and a or a and A)

where: A – experimental (“treatment”) group B – control (“no treatment,” “nonmanipulated”) group

Variable... …a concept (e.g., intelligence, height, aptitude) that can assume any one of a range of values

Research variables... Independent… …an activity of characteristic believed to make a difference with respect to some behavior …(syn.) experimental variable, active variable, cause, treatment

Dependent… …the change or difference occurring a a result of the independent variable …(syn.) criterion variable, assigned variable, effect, outcome, posttest

Confounding… …the fact that the effects of the independent variable may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable …(syn.) criterion variable, assigned variable, effect, outcome, posttest

Group experimental designs… 1. single-variable 2. factorial

single-variable designs …involve one manipulated independent variable pre-experimental quasi-experimental true experimental

types of pre-experimental designs one-shot case study X O …a single group exposed to a treatment ( X ) and then posttested ( O )

one-group pretest-posttest design O X O …a single group is pretested ( O ), exposed to a treatment ( X ) and, then, is posttested ( O )

static group comparison X1 O X2 O …involves at least two groups ( X ), one receiving a new, or experimental treatment ( X1 ) and another receiving a traditional, or control treatment ( X2 ) and, then, are posttested ( O )

types of quasi-experimental designs nonequivalent control group design O X O …random assignment of intact groups that are pretested ( O ), exposed to a treatment ( X ) and then posttested ( O )

time-series design O O O O X O O O O …a single group is pretested ( O ) repeatedly until pretest scores are stable, exposed to a treatment ( X ) and, then, is repeatedly posttested ( O )

counterbalanced design X1 O X2 O X3 O X3 O X1 O X2 O X2 O X3 O X1 O …all of the groups receive all treatments but in a different order; the number of groups and treatments must be equal

types of true experimental designs pretest-posttest control group design R O X1 O R O X2 O

…at least two groups are formed by random assignment ( R ), administered a pretest ( O ), receive different treatments ( X1, X2 ), are administered a posttest, and posttest scores are compared to determine effectiveness of treatments

posttest-only control group design R X1 O R X2 O

…at least two groups are formed by random assignment ( R ), receive different treatments ( X1, X2 ), are administered a posttest, and posttest scores are compared to determine effectiveness of treatments

Solomon four-group design R O X1 O R O X2 O R X1 O R X2 O

…four groups are formed by random assignment ( R ) of participants, two groups are pretested ( O ) and two are not, one pretested and one unpretested group receive the experimental treatments ( X1, X2 ), each group is are administered a posttest on the dependent variable, and posttest scores are compared to determine effectiveness of treatments

factorial designs …involve two or more independent variables with at least one independent variable being manipulated by the researcher

examples of factorial designs two-by-two factorial design (four cells) 2 X 2 …two types of factors (e.g., method of instruction) each of which has two levels (e.g., traditional vs. innovative)

A 2 X 2 factorial design… Independent Variable A B manipulated not manipulated O Group #1 Group #2 Dependent Variable O Group #3 Group #4 Cells

A 2 X 2 factorial design… A No interaction between factors B

A 2 X 2 factorial design… No interaction between factors A B

A 2 X 2 factorial design… A Interacting factors B

A 2 X 2 factorial design… B Interacting factors A

two-by-three factorial design (six cells) 2 X 3 …two types of factors (e.g., motivation; interest) each of which has three levels (e.g., high, medium, low)

Single-subject experimental designs… 1. A – B – A withdrawal 2. multiple baseline designs 3. alternating treatments designs

simple A – B design …baseline measurements ( O ) are repeatedly made until stability is established, then the treatment ( X ) is introduced and an appropriate number of measurements ( O ) are made during treatment implementation

simple A – B design O O O X O X O X O baseline treatment phase phase A | B

A – B – A withdrawal designs …baseline measurements ( O ) are repeatedly made until stability is established, then the treatment ( X ) is introduced and an appropriate number of measurements ( O ) are made during treatment implementation, followed by an appropriate number of baseline measurements ( O ) to determine stability of treatment ( X )

A – B – A withdrawal designs O O O X O X O X O O O baseline treatment baseline phase phase phase A | B | A

multiple-baseline designs …used when a return to baseline conditions is difficult or impossible since treatment effects oftentimes do not disappear when a treatment is removed

…“multiple” refers to the study of more than one behavior, participant, or setting

…instead of collecting baseline data on one specific behavior, data are collected on: (1) several behaviors for one participant, (2) one behavior for several participants, or (3) one behavior and one participant in several settings

…then, over a period of time, the treatment is systematically applied to each behavior (or participant, or setting) one at a time until all behaviors (or participants or settings) have been exposed to the treatment

multiple baseline design Example: one treatment for three behaviors in three settings A B behavior 1 O O OXOXOXOXOXOXOXOXOXOXOXO setting #1 behavior 2 O O O O O OXOXOXOXOXOXOXO setting #2 behavior 3 O O O O O O O O OXOXOXO setting #3 …while the treatment is applied at other settings the baseline remains the same…

alternating treatments design …the relatively rapid and random alternation of treatments ( T1 T2 …Tn ) for a single subject but not necessarily within fixed intervals of time T1 - T2 - T1 - T1 - T2 - T1 - T1 - T2

The concept of validity… …the experiment tests the variable(s) that it purports to test

Threats to validity… …internal: factors other than the independent variable that affect the dependent variable …external: factors that affect the generalizability of the study to groups and settings beyond those of the experiment

Threats to internal validity… 1. history 2. maturation 3. testing 4. instrumentation 5. statistical regression 6. differential selection of participants 7. mortality 8. selection-maturation interaction

history …the occurrence of events that are not part of the experimental treatment but that occur during the study and affect the dependent variable

Educational Research: Experimental Studies