330 likes | 1.2k Views
Quasi-Experimental Design. Jung Eun (Jessie) Hong Feb. 23, 2009. Outlines. Experimental Design Definition Process A Key Point Types of Experimental Designs Quasi-Experimental Design Designs Strengths Weaknesses Examples in Geography Ongoing Debate. Definition of Experimental Design.
E N D
Quasi-Experimental Design Jung Eun (Jessie) Hong Feb. 23, 2009
Outlines • Experimental Design • Definition • Process • A Key Point • Types of Experimental Designs • Quasi-Experimental Design • Designs • Strengths • Weaknesses • Examples in Geography • Ongoing Debate
Definition of Experimental Design • A structured, organized method • To determine whether some program or treatment causes some outcome or outcomes to occur. • If X, then Y • Because there may be lots of reasons, other than the program, for why you observed the outcome, • If not X, then not Y needs to be addressed, too
Process of Experimental Design • To show that there is a casual relationship, • Two “equivalent” groups • The program or treatment group gets the program • The comparison or control group does not • The groups are treated the same in all other respects • Differences in outcomes between two groups must be due to “the program”
A Key Point of Experimental Design • How do we create two groups that are “equivalent”? • Assign people randomly from a common pool of people into the two groups • The experiment relies on the idea of “random assignment” to obtain two similar groups. • A key to the success of the experiment • Assume that two groups are “probabilistically equivalent”
Types of Designs Is random assignment used? No Yes Randomized or True experiment Is there a control group or multiple measures? No Yes Non-experiment Quasi-experiment
Quasi-Experimental Design • Similar to the experimental design, but lacks the key ingredient, “random assignment” • Easily and more frequently implemented • Extensively used in the social sciences • A useful method for measuring social variables • Two classic quasi-experimental designs • The Nonequivalent Groups Design • The Regression-Discontinuity Design
The Nonequivalent Groups Design • The most frequently used in social research • Try to select groups that are as similar as possible to compare the treated one with the comparison one • e.g. two comparable classrooms or schools • Cannot be sure whether the groups are comparable • The groups may be different prior to the study • Any prior differences between the groups may affect the outcome of the study • Require a pretest and posttest
The Regression-Discontinuity Design • A useful method for determining whether a program of treatment is effective • Participants are assigned to program or comparison groups based on a cutoff score on a pretest • e.g. Evaluating new learning method to children who obtained low scores at the previous test. • Cutoff score = 50 • The treatment group: children who obtained 0 to 50 • The comparison group: children who obtained 51 to 100 • The program (treatment) can be given to those most in need
The Regression-Discontinuity Design With no treatment effect With Ten point treatment effect
The Regression-Discontinuity Design Discontinuity
Strengths of Quasi-Experimental Design • Useful in generating results for general trends in social sciences • Difficult pre-selection and randomization of groups • Easily integrated with individual case studies • Generated results can reinforce the findings in a case study • Allow statistical analysis to take place • Enable to reduce the time and resources required for experimentation • Not required extensive pre-screening and randomization
Weaknesses of Quasi-Experimental Design • Without proper randomization, statistical tests can be meaningless • Do not explain any pre-existing factors and influences outside of the experiment • The researcher needs to control additional factors that may have affected the results • Some form of pre-testing or random selection may be necessary to explain statistical results thoroughly
Quasi-experiments vs. Non-experiments to address similar questions • Both designs are applicable when the subjects are not able to be randomized • Some variables cannot ethically be randomized • e.g. Studying the effect of maternal alcohol use when the mother is pregnant
Example of Quasi-Experimental Design in Geography • Baker and White (2003) • The Effects of GIS on Students’ Attitudes, Self-efficacy, and Achievement in Middle School Science Classrooms • Conducted the Nonequivalent quasi-experimental design • Two eighth grade teachers, across ten classrooms • Total 192 eighth grade students participated • Treatment group: used a Web-based GIS application • Control group: used paper maps
Example of Quasi-Experimental Design in Geography • Impossible to randomly assign each student to a GIS or paper mapping conditions • Randomly assigned whole classes to two conditions • Different instructors affected the results differently • Instructor effect played a substantial role in student attitudes and self-efficacy
Ongoing Debate • Whether true experiments or quasi-experiments represents the superior design • Supporters of true experiments • Difficult to isolate the program effects using quasi-experiments • Quasi-experimental results are biased and sensitive to minor changes • Not sure about whether quasi-experimental designs can adequately control selection bias • Hard to determine better design • True experiments are impossible and impractical in some cases