Probability Sampling

1. Probability Sampling • uses random selection • N = number of cases in sampling frame • n = number of cases in the sample • NCn = number of combinations of n from N • f = n/N = sampling fraction

2. Variations • Simple random sampling • based on random number generation • Stratified random sampling • divide pop into homogenous subgroups, then simple random sample w/in • Systematic random sampling • select every kth individual (k = N/n) • Cluster (area) random sampling • randomly select clusters, sample all units w/in cluster • Multistage sampling • combination of methods

3. Nonprobability sampling • accidental, haphazard, convenience sampling ... • may or may not represent the population well

4. Measurement • ... topics in measurement that we don’t have time to cover ...

5. Research Design • Elements: • Samples/Groups • Measures • Treatments/Programs • Methods of Assignment • Time

6. Internal validity • the approximate truth about inferences regarding cause-effect (causal) relationships • can observed changes be attributed to the program or intervention and NOT to other possible causes (alternative explanations)?

7. Establishing a Cause-Effect Relationship • Temporal precedence • Covariation of cause and effect • if x then y; if not x then not y • if more x then more y; if less x then less y • No plausible alternative explanations

8. Single Group Example • Single group designs: • Administer treatment -> measure outcome X -> O • assumes baseline of “0” • Measure baseline -> treat -> measure outcome 0 X -> O • measures change over baseline

9. Single Group Threats • History threat • a historical event occurs to cause the outcome • Maturation threat • maturation of individual causes the outcome • Testing threat • act of taking the pretest affects the outcome • Instrumentation threat • difference in test from pretest to posttest affects the outcome • Mortality threat • do “drop-outs” occur differentially or randomly across the sample? • Regression threat • statistical phenomenon, nonrandom sample from population and two imperfectly correlated measures

10. Addressing these threats • control group + treatment group • both control and treatment groups would experience same history and maturation threats, have same testing and instrumentation issues, similar rates of mortality and regression to the mean

11. Multiple-group design • at least two groups • typically: • before-after measurement • treatment group + control group • treatment A group + treatment B group

12. Multiple-Group Threats • internal validity issue: • degree to which groups are comparable before the study • “selection bias” or “selection threat”

13. Multiple-Group Threats • Selection-History Threat • an event occurs between pretest and posttest that groups experience differently • Selection-Maturation Threat • results from differential rates of normal growth between pretest and posttest for the groups • Selection-Testing Threat • effect of taking pretest differentially affects posttest outcome of groups • Selection-Instrumentation Threat • test changes differently for the two groups • Selection-Mortality Threat • differential nonrandom dropout between pretest and posttest • Selection-Regression Threat • different rates of regression to the mean in the two groups (if one is more extreme on the pretest than the other)

14. Social Interaction Threats • Problem: • social pressures in research context can lead to posttest differences that are not directly caused by the treatment • Solution: • isolate the groups • Problem: in many research contexts, hard to randomly assign and then isolate

15. Types of Social Interaction Threats • Diffusion or Imitation of Treatment • control group learns about/imitates experience of treatment group, decreasing difference in measured effect • Compensatory Rivalry • control group tries to compete w/treatment group, works harder, decreasing difference in measured effect • Resentful Demoralization • control group discouraged or angry, exaggerates measured effect • Compensatory Equalization of Treatment • control group compensated in other ways, decreasing measured effect

16. Intro to Design/ Design Notation • Observations or Measures • Treatments or Programs • Groups • Assignment to Group • Time

17. Observations/Measure • Notation: ‘O’ • Examples: • Body weight • Time to complete • Number of correct response • Multiple measures: O1, O2, …

18. Treatments or Programs • Notation: ‘X’ • Use of medication • Use of visualization • Use of audio feedback • Etc. • Sometimes see X+, X-

19. Groups • Each group is assigned a line in the design notation

20. Assignment to Group • R = random • N = non-equivalent groups • C = assignment by cutoff

21. Time • Moves from left to right in diagram

22. Types of experiments • True experiment – random assignment to groups • Quasi experiment – no random assignment, but has a control group or multiple measures • Non-experiment – no random assignment, no control, no multiple measures

23. Design Notation Example Pretest-posttest treatment versus comparison group randomized experimental design

24. Design Notation Example Pretest-posttest Non-Equivalent Groups Quasi-experiment

25. Design Notation Example Posttest Only Non-experiment

26. Goals of design .. • Goal:to be able to show causality • First step: internal validity: • If x, then y AND • If not X, then not Y

27. Two-group Designs • Two-group, posttest only, randomized experiment Compare by testing for differences between means of groups, using t-test or one-way Analysis of Variance(ANOVA) Note: 2 groups, post-only measure, two distributions each with mean and variance, statistical (non-chance) difference between groups

28. To analyze … • What do we mean by a difference?

29. Possible Outcomes:

30. Three ways to estimate effect • Independent t-test • One-way Analysis of Variance (ANOVA) • Regression Analysis (most general) • equivalent

31. The t-test • appropriate for posttest-only two-group randomized experimental design • See also: paired student t-test for other situations.

32. Measuring Differences …

33. Computing the t-value

34. Computing standard deviation • standard deviation is the square root of the sum of the squared deviations from the mean divided by the number of scores minus one • variance is the square of the standard deviation

35. ANOVA • One-way analysis of variance

36. ANOVA • Analysis of variance – tests hypotheses about differences between two or more means • Could do pairwise comparison using t-tests, but can lead to true hypothesis being rejected (Type I error) (higher probability than with ANOVA)

37. Between-subjects design • Example: • Effect of intensity of background noise on reading comprehension • Group 1: 30 minutes reading, no background noise • Group 2: 30 minutes reading, moderate level of noise • Group 3: 30 minutes reading, loud background noise

38. Experimental Design • One factor (noise), three levels(a=3) • Null hypothesis: 1 =2 =3

39. Notation • If all sample sizes same, use n, and total N = a * n • Else N = n1 + n2 +n3

40. Assumptions • Normal distributions • Homogeneity of variance • Variance is equal in each of the populations • Random, independent sampling • Still works well when assumptions not quite true(“robust” to violations)

41. ANOVA • Compares two estimates of variance • MSE – Mean Square Error, variances within samples • MSB – Mean Square Between, variance of the sample means • If null hypothesis • is true, then MSE approx = MSB, since both are estimates of same quantity • Is false, the MSB sufficiently > MSE

42. MSE

43. MSB • Use sample means to calculate sampling distribution of the mean, = 1

44. MSB • Sampling distribution of the mean * n • In example, MSB = (n)(sampling dist) = (4) (1) = 4

45. Is it significant? • Depends on ratio of MSB to MSE • F = MSB/MSE • Probability value computed based on F value, F value has sampling distribution based on degrees of freedom numerator (a-1) and degrees of freedom denominator (N-a) • Lookup up F-value in table, find p value • For one degree of freedom, F == t^2

46. Factorial Between-Subjects ANOVA, Two factors • Three significance tests • Main factor 1 • Main factor 2 • interaction

47. Example Experiment • Two factors (dosage, task) • 3 levels of dosage (0, 100, 200 mg) • 2 levels of task (simple, complex) • 2x3 factorial design, 8 subjects/group

48. Summary table SOURCE df Sum of Squares Mean Square F p Task 1 47125.3333 47125.3333 384.174 0.000 Dosage 2 42.6667 21.3333 0.174 0.841 TD 2 1418.6667 709.3333 5.783 0.006 ERROR 42 5152.0000 122.6667 TOTAL 47 53738.6667 • Sources of variation: • Task • Dosage • Interaction • Error

49. Results • Sum of squares (as before) • Mean Squares = (sum of squares) / degrees of freedom • F ratios = mean square effect / mean square error • P value : Given F value and degrees of freedom, look up p value

50. Results - example • Mean time to complete task was higher for complex task than for simple • Effect of dosage not significant • Interaction exists between dosage and task: increase in dosage decreases performance on complex while increasing performance on simple