Non-Experimental designs

1. Non-Experimental designs Psych 231: Research Methods in Psychology

2. Sometimes you just can’t perform a fully controlled experiment • Because of the issue of interest • Limited resources (not enough subjects, observations are too costly, etc). • Surveys • Correlational • Quasi-Experiments • Developmental designs • Small-N designs • This does NOT imply that they are bad designs • Just remember the advantages and disadvantages of each Non-Experimental designs

3. Used to study changes in behavior that occur as a function of age changes • Age typically serves as a quasi-independent variable • Three major types • Cross-sectional • Longitudinal • Cohort-sequential Developmental designs

4. Cross-sectional design • Groups are pre-defined on the basis of a pre-existing variable • Study groups of individuals of different ages at the same time • Use age to assign participants to group • Age is subject variable treated as a between-subjects variable Age 4 Age 7 Age 11 Developmental designs

5. Advantages: • Can gather data about different groups (i.e., ages) at the same time • Participants are not required to commit for an extended period of time • Cross-sectional design Developmental designs

6. Disavantages: • Individuals are not followed over time • Cohort (or generation) effect: individuals of different ages may be inherently different due to factors in the environment • Are 5 year old different from 15 year olds just because of age, or can factors present in their environment contribute to the differences? • Imagine a 15yr old saying “back when I was 5 I didn’t have a Wii, my own cell phone, or a netbook” • Does not reveal development of any particular individuals • Cannot infer causality due to lack of control • Cross-sectional design Developmental designs

7. Follow the same individual or group over time • Age is treated as a within-subjects variable • Rather than comparing groups, the same individuals are compared to themselves at different times • Changes in dependent variable likely to reflect changes due to aging process • Changes in performance are compared on an individual basis and overall • Longitudinal design time Age 11 Age 15 Age 20 Developmental designs

8. Example • Wisconsin Longitudinal Study(WLS) • Began in 1957 and is still on-going (50 years) • 10,317 men and women who graduated from Wisconsin high schools in 1957 • Originally studied plans for college after graduation • Now it can be used as a test of aging and maturation Longitudinal Designs

9. Advantages: • Can see developmental changes clearly • Can measure differences within individuals • Avoid some cohort effects (participants are all from same generation, so changes are more likely to be due to aging) • Longitudinal design Developmental designs

10. Disadvantages • Can be very time-consuming • Can have cross-generational effects: • Conclusions based on members of one generation may not apply to other generations • Numerous threats to internal validity: • Attrition/mortality • History • Practice effects • Improved performance over multiple tests may be due to practice taking the test • Cannot determine causality • Longitudinal design Developmental designs

11. Measure groups of participants as they age • Example: measure a group of 5 year olds, then the same group 10 years later, as well as another group of 5 year olds • Age is both between and within subjects variable • Combines elements of cross-sectional and longitudinal designs • Addresses some of the concerns raised by other designs • For example, allows to evaluate the contribution of cohort effects • Cohort-sequential design Developmental designs

12. Cohort-sequential design Time of measurement 1975 1985 1995 Cohort A 1970s Age 5 Age 5 Age 5 Cross-sectional component Cohort B 1980s Age 15 Age 15 Cohort C 1990s Age 25 Longitudinal component Developmental designs

13. Advantages: • Get more information • Can track developmental changes to individuals • Can compare different ages at a single time • Can measure generation effect • Less time-consuming than longitudinal (maybe) • Disadvantages: • Still time-consuming • Need lots of groups of participants • Still cannot make causal claims • Cohort-sequential design Developmental designs

14. Sometimes you just can’t perform a fully controlled experiment • Because of the issue of interest • Limited resources (not enough subjects, observations are too costly, etc). • Surveys • Correlational • Quasi-Experiments • Developmental designs • Small-N designs • This does NOT imply that they are bad designs • Just remember the advantages and disadvantages of each Non-Experimental designs

15. What are they? • Historically, these were the typical kind of design used until 1920’s when there was a shift to using larger sample sizes • Even today, in some sub-areas, using small N designs is common place • (e.g., psychophysics, clinical settings, expertise, etc.) Small N designs

16. One or a few participants • Data are typically not analyzed statistically; rather rely on visual interpretation of the data • Observations begin in the absence of treatment (BASELINE) • Then treatment is implemented and changes in frequency, magnitude, or intensity of behavior are recorded Small N designs

17. Baseline experiments – the basic idea is to show: • when the IV occurs, you get the effect • when the IV doesn’t occur, you don’t get the effect (reversibility) • Before introducing treatment (IV), baseline needs to be stable • Measure level and trend Small N designs

18. Level – how frequent (how intense) is behavior? • Are all the data points high or low? • Trend – does behavior seem to increase (or decrease) • Are data points “flat” or on a slope? Small N designs

19. ABA design (baseline, treatment, baseline) • The reversibility is necessary, otherwise • something else may have caused the effect • other than the IV (e.g., history, maturation, etc.) ABA design

20. Advantages • Focus on individual performance, not fooled by group averaging effects • Focus is on big effects (small effects typically can’t be seen without using large groups) • Avoid some ethical problems – e.g., with non-treatments • Allows to look at unusual (and rare) types of subjects (e.g., case studies of amnesics, experts vs. novices) • Often used to supplement large N studies, with more observations on fewer subjects Small N designs

21. Disadvantages • Effects may be small relative to variability of situation so NEED more observation • Some effects are by definition between subjects • Treatment leads to a lasting change, so you don’t get reversals • Difficult to determine how generalizable the effects are Small N designs

22. Some researchers have argued that Small N designs are the best way to go. • The goal of psychology is to describe behavior of an individual • Looking at data collapsed over groups “looks” in the wrong place • Need to look at the data at the level of the individual Small N designs