1 / 30

Validity, Sampling & Experimental Control

Validity, Sampling & Experimental Control. Psych 231: Research Methods in Psychology. Announcements. The required articles for the class experiment paper are already on-line at the Milner course reserves page. Class Experiment.

aiko
Download Presentation

Validity, Sampling & Experimental Control

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Validity, Sampling & Experimental Control Psych 231: Research Methods in Psychology

  2. Announcements • The required articles for the class experiment paper are already on-line at the Milner course reserves page

  3. Class Experiment • Collect the forms (consent forms and data summary sheets) - pass to the front • Brief discussion: • So how did it go? • What happened? • Any thing unusual/unexpected? • Any problems?

  4. External Validity • Are experiments “real life” behavioral situations, or does the process of control put too much limitation on the “way things really work?” • Will the same basic conclusions be supported with different operational definitions, different participants, different research settings?

  5. External Validity • Variable representativeness • relevant variables for the behavior studied along which the sample may vary • Subject representativeness • characteristics of sample and target population along these relevant variables • Setting representativeness (ecological validity) • how do the characteristics of the research setting compare with the “real world”

  6. Internal Validity • The precision of the results • Did the change result from the changes in the DV or does it come from something else?

  7. Threats to internal validity • History – an event happens during the experiment • Maturation – participants get older (and other changes) • Selection – nonrandom selection may lead to biases • Mortality – participants drop out or can’t continue • Testing – being in the study actually influences how the participants respond • Statistical regression – regression towards the mean, if you select participants based on high (or low) scores (e.g., IQ, SAT, etc.) their scores later tend to move towards the mean.

  8. “Debugging your study” • Pilot studies • A trial run through • Don’t plan to publish these results, just try out the methods • Manipulation checks • An attempt to directly measure whether the IV variable really affects the DV. • Look for correlations with other measures of the desired effects.

  9. Sampling • Why do we do we use sampling methods? • Typically don’t have the resources to test everybody • Population - everybody that the research results are targeted • Sample - the subset of the population that actually participates in the research

  10. Sampling • Goals: • Maximize: • Representativeness - to what extent do the characteristics of those in the sample reflect those in the population • Reduce: • Bias - a systematic difference between those in the sample and those in the population

  11. Sampling Methods • Probability sampling • Simple random sampling • Systematic sampling • Stratified sampling • Non-probability sampling • Convenience sampling • Quota sampling

  12. Simple random sampling • Every individual has a equal and independent chance of being selected from the population

  13. Systematic sampling • Selecting every nth person

  14. Stratified sampling • Step 1: Identify groups (strata) • Step 2: randomly select from each group

  15. Convenience sampling • Use the participants who are easy to get

  16. Quota sampling • Step 1: identify the specific subgroups • Step 2: take from each group until desired number of individuals

  17. Experimental Control • Our goal: • to test the possibility of a relationship between the variability in our IV and how that affects our DV. • Control is used to minimize excessive variability. • To reduce the potential of confoundings. • if there are other variables that influence our DV, how do we know that the observed differences are due to our IV and not some other variable

  18. Sources of variability (noise) • Sources of Total (T) Variability: T = NonRandomexp + NonRandomother +Random

  19. Sources of variability (noise) I. Nonrandom (NR) Variability – systematic variation A. (NRexp)manipulated independent variables (IV) i. our hypothesis is that changes in the IV will result in changes in the DV B. (NRother)extraneous variables (EV) which covary with IV i. other variables that also vary along with the changes in the IV, which may in turn influence changes in the DV

  20. Sources of variability (noise) II. Random (R) Variability A. imprecision in manipulation (IV) and/or measurement (DV) B. randomly varying extraneous variables (EV)

  21. Sources of variability (noise) • Sources of Total (T) Variability: T = NRexp + NRother +R • Our goal is to reduce R and NRother so that we can detect NRexp. • That is, so we can see the changes in the DV that are due to the changes in the independent variable(s).

  22. NR NR NR R R other other exp Weight analogy • Imagine the different sources of variabilility as weights Treatment group control group

  23. NR NR NR R R other other exp Weight analogy • If NRother and R are large relative to NRexp then detecting a difference may be difficult

  24. NR NR NR R R other other exp Weight analogy • But if we reduce the size of NRother and R relative to NRexp then detecting gets easier

  25. Methods of Controlling Variability • Comparison • Production • Constancy/Randomization

  26. Methods of Controlling Variability • Comparison • An experiment always makes a comparison, so it must have at least two groups • Sometimes there are baseline, or control groups • This is typically the absence of the treatment • Without control groups if is harder to see what is really happening in the experiment • it is easier to be swayed by plausibility or inappropriate comparisons • Sometimes there are just a range of values of the IV

  27. Methods of Controlling Variability • Production • The experimenter selects the specific values of the Independent Variables • (as opposed to allowing the levels to freely vary as in observational studies) • Need to do this carefully • Suppose that you don’t find a difference in the DV across your different groups • Is this because the IV and DV aren’t related? • Or is it because your levels of IV weren’t different enough

  28. Methods of Controlling Variability • Constancy/Randomization • If there is a variable that may be related to the DV that you can’t (or don’t want to) manipulate • Then you should either hold it constant, or let it vary randomly across all of the experimental conditions

  29. Potential Problems of Experimental Control • Excessive random variability: • If control procedures are not applied, then R component of data will be excessively large, and may make NR undetectable • Confounding: • If relevant EV covaries with IV, then NR component of data will be "significantly" large, and may lead to misattribution of effect to IV • Dissimulation: • If EV which interacts with IV is held constant, then effect of IV is known only for that level of EV, and may lead to overgeneralization of IV effect

  30. Next time • Read: Chpt 8

More Related