Research Validity

1. 6 Research Validity

2. Research Validity • Research validity refers to the correctness or truthfulness of an inference that is made from the results of a research study. • Four Major Types of Validity • Statistical conclusion validity: Extent to which IV affects the DV…. There is a change & whether results are statistically significant • Construct validity: How valid is our construct or measurements used to measure it. Depression is a CONSTRUCT… • How do you define “Diversity? • Internal validity • External validity

3. Statistical Conclusion Validity • Do independent and dependent variables covary? • ANY VARIATION OF IV CHANGES DV • Inferential statistics allow us to establish this type of validity • Small sample size is a threat to statistical conclusion validity • NOT ENOUGH power (chapter 9: Power Analysis) • LIKELY to conclude that there is no effect, when in fact there is a difference (Commit type II Error)… • Have large enough sample to find a difference if there is one

4. Construct Validity • Construct validity—extent to which we can infer higher-order constructs for our operations • Diversity: Number of racial-ethnic groups in the community & the population & The sizes of the groups relative to each other (e.g., a Population consisting of many groups of equal size 4 would be highly diverse (Lee, Iceland, & Sharp, 2012) • Entropy (E) = gauges how uniformly members of a population are spread. E = 0 signifies complete homogeneity or no diversity; E =100 indicates maximum heterogeneity proportions of the population • Constructs are used for • research participants: How we classify them: MF, Hispanic,Mex • independent variable: We choose the levels… we name them • dependent variable” We choose the measurement • experimental setting: Experimental vs. nonexperimental,

5. Entropy index (E): Index gauges how uniformly members of a population are spread. E = 0 = complete homogeneity or no diversity

7. Threats to Construct Validity (cont'd) • Reactivity to the Experimental Situation • research participants’ motives and perceptions influencing their response to the DV • motive and perceptions influenced by the demand characteristics of the experiment (cues available… experiment, what they heard) • primary motive--positive self-presentation: Present themselves in the most positive manner.. • condition producing positive self-presentation motive • If participant’s believe that others view their behavior as under the internal control of each participant Positive self-presentation • If participants believe that others view their behavior as being determined by external source NOT under their control… No Positive self-presentation

8. Threats to Construct Validity (cont'd) • Experimenter Effect • Actions and characteristics of researchers that influence the responses of participants • Can be intentional or unintentional • Experimenter’s motive of supporting the study hypothesis can lead to bias • Clever Hans: The Horse that solved arithmetic operations • Biases communicated to the participants… (demand characteristics)

9. Threats to Construct Validity (cont'd) • Ways experimenter may bias the study • experimenter attributes: • Biasing experimenter effects attributable to the physical & psychological characteristics of the researcher • Biosocial: Gender, age, religion • Psychosocial: authoritative, anxious • Situational factors: setting of the experiment, experiments naive, or experienced • experimenter expectancies: Attributed to expected outcomes of the experiment, biasing subjects to answer according to expectations. • effect on experimenter—recording bias • effect on research participant— • nonverbal communication in human studies • E.g., experimenter acts differently toward experimental vs. control to influence study to support hypothesis

11. Internal Validity • THE CORRECTNESS of INFERENCES MADE BY RESEARCHERS ABOUT CAUSE AND EFFECT • Criteria for identifying a causal relationship • Relationship condition: IV related to DV • Temporal order condition: Changes in IV Precede Changes in DV • No other plausible explanation must exist for the effect • Primary threat is confounding extraneous variables • Eliminate the confounding influence of extraneous variables by holding their influence constant • Randomization, equate on intelligence, do experiments not very late.. Avoid.. very young and very old subjects (unless variable of interest).

12. Threats to Internal Validity • History - any event that occurs between the pretest and posttest that can produce the outcome • Schoenthaler (1983): Dietary and Violent Behaviors (Juveniles) • PRE (very aggressive Behaviors for 3 months) • POST (decrease aggressive behaviors) • Six months between pretest vs. posttest (Changes in policies)? • IMPROVED THE STUDY? USE EXPERIMENTAL GROUP • OTHER EXAMPLES? STUDIES OF ARAB AMERICANS? • differential history occurs in multigroupdesign when event has differential impact on groups DIFFERENT HISTORY EVENTS) • Woman in the experimental group Women, Infants, Children (WIC) • Experimental WIC + Also qualified for FOOD STAMPS (new policy) • Control Group only qualified for WIC (Shadish & Reis (1984)

13. History • Example • a researcher wants to test a new treatment for bipolar disorder on a group of patients. All patients seem to be showing improvement after 4 weeks • history threat – at Week 2 of treatment, actress Catherine Zeta Jones announces that she is bipolar and wants everyone who is also suffering with it to know that treatment is key • can the researcher conclude that the new treatment is responsible for the improvement in the patients?

14. Differential history • Example • Shadish and Reid (1984) • evaluation of the efficacy of the WIC program • experimental group – received WIC assistance • control group – did not receive WIC assistance • differential history threat – women who received WIC also received food stamps • can you conclude that the WIC assistance led to improved pregnancy outcomes in the experimental group?

16. Threats to Internal Validity • Maturation – internal changes of research participants that occur over time (Hunger, Age, Fatigue, Learning, Boredom) • HEAD START PROGRAM (TO SEE ACADEMIC IMPROVEMENTS) • PRETEST: ACHIEVEMENT TEST (SCORE OF 50) • POSTTEST: ACHIEVEMENT TEST (SCORE OF 70) GOOD RESULTS? NO! BECAUSE WE NEED CONTROL GROUP: IMPROVEMENT COULD BE DUE TO CHANGE THAT OCCUR OVER TIME!! (Mental maturation) typically can be controlled for with comparison control group

17. Threats to Internal Validity (cont'd) • Instrumentation – changes in the measurement of the dependent variable • e.g., if human observers change measurement because they become bored or fatigued • Observers not very good at testing and become better over time.. More accurate and less mistakes.. More skilled… • THINK ABOUT BOXING? And judges… Too subjective? • USE COMPUTERIZED INSTRUMENTS • using multiple observers can assess the validity and reliability of their observations • training observers on observation techniques can also decrease instrumentation bias • Testing – occurs when the influence of taking the pretest affects the posttest • changes in a person’s score on the second administration of a test resulting from having previously taken the test

18. Threats to Internal Validity (cont'd) • Regression Artifact (Regression toward the mean)– the tendency for extreme scores to be closer to average at posttest • potential problem if participants with extreme scores at pretest are selected for study • Select Highly Depressed individuals: After one treatment High Scores.. .close to the mean! • EXAM1: In class multiple Choice: VERY LOW SCORES • EXAM 2: on-line Multiple Choice: VERY LOW SCORES NOW HIGH • EXAM 3: In class multiple Choice: GO BACK TO LOW SCORES • ALWAYS USE CONTROL GROUP TO AVOID THESE ISSUES • Attrition – drop out rate of research participants • a potential threat in two group designs where differential attrition occurs

19. Threats to Internal Validity (cont'd) • Selection – potential threat in a two group design when different selection procedures are used • That is why we randomly assign AND RANDOMLY SELECT • Nonequivalent groups: One group containing more females than males... (and differences due to gender!) • Additive and Interactive effects – produced by the combined effect of two or more threats • Selection-history: effect occurs when the groups are exposed to the same history but react to it differently • Selection-Maturation: Effect occurs if the groups mature at different rates… • e.g., comparing 6 year-old vs. 10 year-olds. Six year olds… improve more on reading.

20. External Validity • Generalizing across people, settings, treatment variations, outcomes and times • A failure to generalize can result from several factors: • lack of random selection • chance variation: REPLICATE • failure to identify interactive effects of independent variables • IF THOSE EFFECTS EXIST! • e.g., the effect of an attitude change procedure interacts with gender, meaning it works better for males compared to females

21. Types of External Validity • Population validity • do results generalize from sample to target population? • REPRESENTATIVE SAMPLE • Ecological validity (MIRRORS THE REAL WORLD) • do results of study generalize to different setting? • common criticism of laboratory experiments

22. Types of External Validity (cont'd) • Outcome validity • do results generalize to other, related, dependent variables? • DO RESULTS REPLICATE OTHER STUDIES USING OTHER DV’S • E.G., • REACTION TIMES VS. % OF ERRORS READING • REACTION TIMES VS. EYE-FIXATIONS? • DO RESULTS CONVERGE?

23. Relationship between Internal and External Validity • Relationship between internal and external validity is often inverse • Factors that increase our ability to establish cause and effect tend to decrease our ability to generalize • External validity is established through replication