Types of research design – experiments

1. Types of research design – experiments • Chapter 8 in Babbie & Mouton (2001) • Introduction to all research designs • All research designs have specific objectives they strive for • Have different strengths and limitations • Have validity considerations Types of design - experiments

2. Validity considerations • When we say that a knowledge claim (or proposition) is valid, we make a JUDGEMENT about the extent to which relevant evidence supports that claim to be true • Is the interpretation of the evidence given the only possible one, or are there other plausible ones? • "Plausible rival hypotheses" = potential alternative explanations/claims • e.g. New York City's "zero tolerance" crime fighting strategy in the 1980s and 1990s - the reverse of the "broken windows" effect Types of design - experiments

3. The logic of causal social research in the controlled experiment • Explanatory rather than descriptive • Different from correlational research - one variable is manipulated (IV) and the effect of that manipulation observed on a second variable (DV) • If … then …. • E.g. • "Animals respond aggressively to crowding" (causal) • "People with premarital sexual experience have more stable marriages" (noncausal) Types of design - experiments

4. Three pairs of components: • Independent and dependent variables • Pre-testing and post-testing • Experimental and control groups Types of design - experiments

5. Components • Variables • Dependent (DV) • Independent (IV) • Pre-testing and post-testing • O X O • Experimental and control groups • To off-set the effects of the experiment itself; to detect effects of the experiment itself Types of design - experiments

6. The generic experimental design: • R O1 X O2 R O3 O4 • The IV is an active variable; it is manipulated • The participants who receive one level of the IV are equivalent in all ways to those who receive other levels of the IV Types of design - experiments

7. Sampling • 1. Selecting subjects to participate in the research • Careful sampling to ensure that results can be generalized from sample to population • The relationship found might only exist in the sample; need to ensure that it exists in the population • Probability sampling techniques Types of design - experiments

8. Sampling • 2. How the sample is divided into two or more groups is important • to make the groups similar when they start off • randomization - equal chance • matching - similar to quota sampling procedures • match the groups in terms of the most relevant variables; e.g. age, sex, and race Types of design - experiments

9. Variations on the standard experimental design • One-shot case study X O • No real comparison Types of design - experiments

10. A famous one-group posttest-only design • Milgram's study on obedience • Obedience to authority • The willingness of subjects to follow E's orders to give painful electrical shocks to another subject • A real, important issue here: how could "ordinary" citizens, like many Germans during the Nazi period, do these incredibly cruel and brutal things? • If a person is under allegiance to a legitimate authority, under what conditions will the person defy the authority if s/he is asked to carry out actions clearly incompatible with basic moral standards? Types of design - experiments

11. One-group pre-test post-test design • O1 X O2 Types of design - experiments

12. Example • We want to find out whether a family literacy programme enhances the cognitive development of preschool-age children. • Find 20 families with a 4-year old child, enrol the family in a high-quality family literacy programme • Administer a pretest to the 20 children - they score a mean of say 50 on the cognitive test • The family participates in the programme for twelve months • Administer a post-test to the 20 children; now they score 75 on the test - a gain of 25 Types of design - experiments

13. Two claims/conclusions: • 1 The children gained 25 points on average in terms of their cognitive performance • 2 the family literacy programme caused the gain in scores • VALIDITY - rival explanations Types of design - experiments

14. Static-group comparison X O O Types of design - experiments

15. Evaluating research (experiments) • We know the structure of research • We understand designs • We know the requirements of "good" research • Then we can evaluate a study • Is it good? Can we believe its conclusions? • Back to plausible rival hypotheses Types of design - experiments

16. Validity in designs • If the design is not valid, then the conclusions drawn are not supported; it is like not doing research at all • Validity of designs come in two parts: • Internal validity • can the design sustain the conclusions? • External validity • can the conclusions be generalized to the population? Types of design - experiments

17. Internal validity • Each design is only capable of supporting certain types of conclusions • e.g. only experiments can support conclusions about causality • Says nothing about if the results can be applied to the real world (generalization) • Generally, the more controlled the situation, the higher the internal validity • The conclusions drawn from experimental results may not accurately reflect hat has gone on in the experiment itself Types of design - experiments

18. Sources of internal invalidity • These sources often discussed as part of experiments, but can be applied to all designs (e.g. see reactivity) • History • Historical events may occur that will be confounded with the IV • Especially in field research (compare the control in a laboratory, e.g. nonsense syllables in memory studies Types of design - experiments

19. Maturation • Changes over time can be caused by a natural learning process • People naturally grow older, tired, bored, over time Types of design - experiments

20. Testing (reactivity) • People realize they are being studied, and respond the way they think is appropriate The very act of studying something may change it • In qualitative research, the "on stage" effects Types of design - experiments

21. The Hawthorne studies • Improved performance because of the researcher's presence - people became aware that they were in an experiment, or that they were given special treatment • Especially for people who lack social contacts, e.g. residents of nursing homes, chronic mental patients Types of design - experiments

22. Placebo effect • When a person expects a treatment or experience to change her/him, the person changes, even when the "treatment" is know to be inert or ineffective • Medical research • "The bedside manner", or the power of suggestion Types of design - experiments

23. Experimenter expectancy • Pygmalion effect - self-fulfilling prophecies of e.g. teachers' expectancies about student achievement • Experimenters may prejudge their results - experimenter bias • Double blind experiments: • Both the researcher and the research participant are "blind" to the purpose of the study. • They don't know what treatment the participant is getting Types of design - experiments

24. Instrumentation • Instruments with low reliability lead to inaccurate findings/missing phenomena • e.g. human observers become more skilled over time (from pretest to posttest) and so report more accurate scores at later time points Types of design - experiments

25. Statistical regression to the mean • Studying extreme scores can lead to inflated differences, which would not occur in moderate scorers Types of design - experiments

26. Selection biases • Selection subjects for the study, and assigning them to E-group and C-group • Look out for studies using volunteers Types of design - experiments

27. Attrition • Sometimes called experimental (or subject) mortality • If subjects drop out, it creates a bias to those who did not • e.g. comparing the effectiveness of family therapy with discussion groups for treatment of drug addiction • addicts with the worst prognosis more likely to drop out of the discussion group • will make it look like family therapy does less well than discussion groups, because the "worst cases" were still in the family therapy group Types of design - experiments

28. Diffusion or imitation of treatments • When subject can communicate to each other, pass on some information about the treatment (IV) Types of design - experiments

29. Compensation • In real life, people may feel sorry for C-group who does not get "the treatment" - try to give them something extra • e.g. compare usual day care for street children with an enhanced day treatment condition • service providers may very well complain about inequity, and provide some enhanced service to the children receiving usual care Types of design - experiments

30. Compensatory rivalry • C-group may "work harder" to compete better with the E-group Types of design - experiments

31. Demoralization • Opposite to compensatory rivalry • May feel deprived, and give up • e.g. giving unemployed high school dropouts a second chance at completing matric via a special education programme • if we assign some of them to a control group, who receive "no treatment", they may very well become profoundly demoralized Types of design - experiments

32. External validity • Can the findings of the study be generalized? • Do they speak only of our sample, or of a wider group? • To what populations, settings, treatment variables (IV's), and measurement variables can the finding be generalized? Types of design - experiments

33. External validity • Mainly questions about three aspects: • Research participants • Independent variables, or manipulations • Dependent variables, or outcomes • Says nothing about the truth of the result that we are generalizing • External validity only has meaning once the internal validity of a study has been established • Internal validity is the basic minimum without which an experiment is uninterpretable Types of design - experiments

34. External validity • Our interest in answering research questions is rarely restricted to the specific situation studied - our interest is in the variables, not the specific details of a piece of research • But studies differ in many ways, even if they study the same variables: • operational definitions of the variables • subject population studied • procedural details • observers • settings • Generally bigger samples with valid measures lead to better external validity Types of design - experiments

35. Sources of external invalidity • Subject selection - Selecting a sample which does not represent the population well, will prevent generalization • Interaction between the testing situation and the experimental stimulus • When people have been sensitized to the issues by the pre-test • Respond differently to the questionnaires the second time (post-test) • Operationalization Types of design - experiments

36. Operationalization • We take a variable with wide scope and operationalize it in a narrow fashion • Will we find the same results with a different operationalization of the same variable? Types of design - experiments

37. Field experiments • "natural" - e.g. disaster research • Static-group comparison type • Non-equivalent experimental and control groups Types of design - experiments

38. Strengths and weaknesses • Strengths • Control • Manipulating the IV • Sorting out extraneous variables • Weaknesses • Articifiality - a generalization problem • Expense • Limited range of questions Types of design - experiments

39. IN CONCLUSION • Donald Campbell often cited Neurath's metaphor: • "in science we are like sailors who must repair a rotting ship while it is afloat at sea. We depend on the relative soundness of all other planks while we replace a particularly weak one. Each of the planks we now depend on we will in turn have to replace. No one of them is a foundation, nor point of certainty, no one of them is incorrigible" Types of design - experiments