Reliability and Validity in Experimental Research ♣

1. Reliability and Validity in Experimental Research♣ Chapter 7 Back to Brief Contents • Introduction  • Relationship between Reliability and Validity  • Experimental Reliability  • Reliability and the Independent Variable  • Reliability and the Dependent Variable  • Measurement Error  • Methods of Assessing Reliability  • Experimental Validity  • Statistic Conclusion Validity  • Internal Validity 

2. 7.0 Introduction -1 Back to Chapter Contents • Reliability • Consistency, stability, or repeatability • Experimental reliability • Consistency, stability, or repeatability of the results of an experimental study

3. 7.0 Introduction -2(end) Back to Chapter Contents • Validity • The correctness or truthfulness of an inference • Experimental validity • The correctness or truthfulness of an inference that is made from the results of an experiment

4. 7.1 Relationship between Reliability and Validity Back to Chapter Contents • Experimental reliability can exist without experimental validity • Experimental validity cannot exist without experimental reliability • (e.g.) speedometer, tires

5. 7.2 Experimental Reliability-1 Back to Chapter Contents • Number of research participants • More Participants  more reliable • Large N: less likely to be influenced by random or chance factors • Reliability of identifying the type of participant needed for the study • Indicators used select P with the same characteristics • (e.g.) Beck Depression Inventory

6. 7.2 Experimental Reliability -2 Back to Chapter Contents • Characteristics of the researchers • (e.g.) sleep deprivation on heart rate: male P, male vs. attractive blond • Maximize reliability: the same researcher collects data from all P • Replication: researcher with the same characteristics should be employed

7. 7.2 Experimental Reliability-3(end) Back to Chapter Contents • Ability of the researcher • Use the same procedure • Well trained  good recording (e.g.) EEG

8. 7.3 Reliability and the Independent Variable Back to Chapter Contents • Consistency: a psychological construct is created in the P • Concrete operations must produce the same effect in P for reliability to exist • (e.g.) Velten (1968): mood state read a set of statements and to feel and place themselves in that situation

9. 7.4 Reliability and the Dependent Variable Back to Chapter Contents • The effect measured should be about the same regardless of when it is measured

10. 7.5 Measurement Error -1 Back to Chapter Contents • Definition — inaccuracy that exists when measuring a psychological characteristic or behavior • Factors causing measurement error— P respond differently on different occasions • (e.g.) fatigue, emotionally upset • Relationship between measurement error and reliability • Measurement error   Reliability 

11. 7.5 Measurement Error -2(end) Back to Chapter Contents • Measurement = true score + error score • Most experiments: single-occasion events • Epstein (1981): low reliability • Averaged over several days • Epstein (1979)Fig 7.1 • Repeat and Average (but most exp. cannot) • Replication of the results of single-occasion studies

12. 7.6 Methods of Assessing Reliability Back to Chapter Contents • Successive measurement • Test-retest, Equivalent forms • Simultaneous measurement • Interrater reliability: Consistency of ratings made by the different observers (continuous v.) • Interobserver agreement: Percentage of times different raters agree (discrete v.) • Internal consistency • Split-half, Cronbach’s alpha

13. 7.7 Experimental Validity Back to Chapter Contents • Definition—accuracy or correctness of an inference • Types of validity that maximize obtaining experimental validity (Shadish et. Al., 2002) Tab 7.1 • Statistical conclusion validity • Internal validity • Construct validity • External validity

14. 7.8 Statistical Conclusion Validity Back to Chapter Contents • Refers to the accuracy of an inference that the IV and DV covary • Threats • insufficient research participants -- power • Many others but beyond this course to discuss

15. 7.9 Internal Validity -1 Back to Chapter Contents • Refers to the accuracy of the inference that the IV caused the effect observed in the DV • Primary threat — confounding extraneous variables (EV) • Confounding: systematically vary with IV • EV: variable other than IV that influences DV

16. 7.9 Internal Validity-2 Back to Chapter Contents • Threats to Internal Validity Shadish, Cook & Campbell (2002) • History • Maturation • Instrumentation • Testing • Regression Artifact • Attrition • Selection • Additive and Interactive Situation

17. 7.9 Internal Validity-3 Back to Chapter Contents • Threats to Internal Validity • History Fig 7.2 Any event occurring after the experimental treatment is introduced that could produce the observed outcome • Maturation Changes in biological and psychological conditions that occur with the passage of time • Instrumentation Changes in the assessment of the DV — human observers: more than one (check) training program

18. 7.9 Internal Validity-4 Back to Chapter Contents • Threats to Internal Validity • Testing Changes in a person’s score on the second administration of a test as a result of previously having taken the test • Regression Artifact Tab 7.2Fig 7.3 • Attrition 耗損 Some people do not show up for the study or do not complete it

19. 7.9 Internal Validity-5(end) Back to Chapter Contents • Threats to Internal Validity • Selection The choice of P for the various treatment groups is based on different criteria (e.g.) type of psychosis • Additive and Interactive Situation The combined effect of several threats to internal validity — selection-maturation effect Fig 7.4 — selection-history effect — selection-instrumentation effect