Hypothesis Testing

1. Hypothesis Testing Chapter 13

2. Hypothesis Testing • Decision-making process • Statistics used as a tool to assist with decision-making • Scientific hypothesis is a statement of the predicted relationship amongst the variables • Null hypothesis is a statement of no relationship amongst the variables

3. Null Hypothesis Not Rejected Total Population Sample reared in sterile environment Sample reared in enriched environment

4. Null Hypothesis Rejected Total population of rats reared in sterile environment Total population of rats reared in enriched environment Sample used in study Sample used in study

5. Hypothesis TestingIn Experimental Studies • Your research design determines the kind of statistical test you will use. • Experimental studies test hypotheses while quasi-experimental studies tend to focus more on generating hypotheses.

6. Research Designs/Approaches

7. Research Designs/Approaches

8. Research Designs/Approaches

9. Research Designs/Approaches

10. Tests of Significance

11. Experimental Designs • Examines differences between experimentally manipulated groups or variables (e.g., one group gets a certain drug and the other gets a placebo). • At minimum, experimental (independent) variable has two levels (e.g., drug vs. placebo). • Advantage is that you can determine causality. • Disadvantage is cost and many variables cannot be experimentally manipulated (e.g., smoke exposure over time).

12. Null HypothesisSignificance Testing • Null hypothesis • Results are due to “chance” • H0 • Alternative (scientific) hypothesis • Results are due to a true “effect” • H1

13. Null HypothesisSignificance Testing • Null hypothesis • Results are due to “chance” (H0) • Alternative (scientific) hypothesis • Results are due to a true “effect” (H1) • Assess • Assuming H0 is true, what is the probability or “chance” of obtaining the data we did?

14. Null HypothesisSignificance Testing • Null hypothesis • Results are due to “chance” (H0) • Alternative (scientific) hypothesis • Results are due to a true “effect” (H1) • Assess • Assuming H0 is true, what is the probability or “chance” of obtaining the data we did? • Decide • If the chance is small enough, reject H0 and infer the “effect” is real.

15. Experimental Designs:Hypothesis Testing

16. Parametric Vs. Non-Parametric Statistics: Two-Sample Cases

17. Parametric Vs. Non-Parametric Statistics: > 2-Sample Cases

18. Parametric Vs. Non-Parametric Statistics: > 2-Sample Cases

19. Sampling Distribution of Mean Difference Scores 95% of all cases 0 99% of all cases

20. Critical Values of T • Need to determine the degrees of freedom • df = N-2 • Need to determine the p value for rejecting the null hypothesis (alpha) • Need to determine if this is a 1-tailed or 2-tailed level of significance.

21. T-Values • T120 = 2.00, p < 0.05

22. What is one of the major criticisms of employing statistical tests of the null hypothesis to determine if effects are true?

23. Limitations of Statistical Tests of the Null Hypothesis • Does not take into account the size of the difference between means (effect size)

24. Analysis of Variance (ANOVA) • F-ratio = MSbet MSwithin • Essentially is the between group variance divided by the within group variance. • If the groups come from similar populations, the variances between the groups will be similar to the variance within groups (null hypothesis is not rejected).

25. ANOVA • Between group variance consists of: • Variability due to the effect of the independent variable (treatment effect) • Variability due to chance factors • Within group variance consists of: • Variability in data with the treatment groups that is due to chance since if treatment effect was consistent, all subjects within a treatment group would experience similar magnitude of effect.

26. Analysis of Variance (ANOVA) • F-ratio = MSbet MSwithin • The MS refers to the mean square and is the sums of squares divided by the appropriate degrees of freedom. • Df for MSbet is the number of groups minus 1. • Df for MSwithin is the total number of scores in the experiment minus the number of groups.

27. ANOVA MSbet = treatment effect + chance variability MSwithin = chance variability • Ratio will be 1 if there is no treatment effect • F(2,144) = 5.56, p < 0.05.

28. Two-Way ANOVA • Where you have 2 independent variables, each having at least 2 levels. For example, • Drug dose (none vs. 5 mg) • Delivery mood (intravenous vs. oral) • Factorial design so you can test both main effects and interaction effects

29. Mixed Model:2 Between Subject Factors1 within Subject Factor • Where you have 2 independent variables, each having at least 2 levels. For example, • Drug dose (none vs. 5 mg) • Delivery mood (intravenous vs. oral) • One within subject factor with for example 3 levels • Pre-treatment, 3 and 6 months follow-up • Factorial design so you can test both main effects and interaction effects (3-way interaction effects)

30. Rejecting the Null Hypothesis • Null hypothesis can be rejected but not accepted • Arguments made for allowing some flexibility in being able to conclude the null hypothesis is true; • No other studies of the phenomenon have rejected the null hypothesis • P value for the test of the null hypothesis is large (e.g., > .20 or .40). • Research design is sufficiently powerful

31. Errors in Statistical Decision-Making • Type I error – falsely reject the null hypothesis • At p < .05 there is a 5% chance (5 in 100) of falsely rejecting null hypothesis • Type II error – failing to reject the null hypothesis when it is false

32. External Validity Chapter 14

33. Goals of Psychology Research • Goal is to understand the underlying laws governing the behaviour of organisms. • The extent to which the results of your study help inform one about these underlying laws, the more valuable the findings. • Limits to the importance of the findings are the internal/external validity.

34. External Validity • Extent to which the results of the study can be generalized across different persons, settings, and times. • Typically think of generalizing to specific populations (e.g., North American elementary school students) than world at large. • Best safeguard is random selection but not usually feasible.

35. Threats to External Validity • Lack of population validity • Lack of ecological validity • Lack of time validity

36. Population Validity • Generalizing to the defined population (i.e., target population) from which the sample was drawn. • Sample is the experimentally accessible population.

37. Target Population Experimentally accessible population Sample Population Validity

38. Population Validity • Threatened by a selection by treatment interaction: • Treatment results may not be exactly reproducible in target population. • Even willingness to volunteer for studies have been shown to result in a selection by treatment interaction effect.

39. Ecological Validity • Extent to which the results can be generalized across settings or environmental conditions. • E.g., Would the treatment effect observed in patients recruited from a 1st class medical centre be the same as the the treatment effect observed in patients recruited from a local community hospital?

40. Ecological Validity • Multiple-Treatment Interference • Sequencing effect whereby exposure to one treatment influences responses to another treatment; or • Exposure to one experiment influences response in another experiment (e.g., sophisticated participants).

41. Ecological Validity • Hawthorne Effect • Knowing one is in a study can affect one’s behaviour • Participant bias effects (e.g., social acceptability, compliance) • Novelty or Disruption Effect • Effects are simply due to novelty and wear off once novelty diminishes.

42. Ecological Validity • Experimenter Effect • Enthusiastic experimenter/clinician may get different effects than a clinician who is implementing the treatment in routine care. • Pre-testing Effect • Administering a pre-test may sensitive the participant in such a way that he/she may respond differently to the experiment than what would have occurred without a pre-test.

43. Temporal Validity • Extent to which the results would generalize to other times • Results might vary depending on the time elapsed between presentation of the independent variable and the measurement of the dependent variable.

44. Temporal Validity • Seasonal Variation • Variation that appears regularly over time (e.g., change in traffic accident rates between daylight savings time and non-daylight savings time). • Fixed-time variation – variation at specific, predictable time points • Variable-time variation – don’t know when variation will occur but when it occurs, there are predictable responses.

45. Temporal Validity • Cyclical Variation • Predictable variation within people or other organisms • Personological Variation • Variation in the characteristics of the individual over time

46. Internal Vs. External Validity • Tends to be an inverse relationship • Internal validity ; external validity • In testing for between group differences, you want to minimize within group variability and maximize between group differences • To do so you want to ensure high control over factors that could confound the results but this often results in increasingly artificial experimental conditions.

47. When Is External Validity Less Important • When you don’t need to demonstrate that “X” will happen but rather “X” can happen. • Sometimes the main goal is to test a theory and extent to which it reflects “real-life” is less important.