1 / 85

Understanding Study Designs: Experimental vs. Quasi-experimental

Explore the differences between experimental and quasi-experimental study designs, including validity types and threats, to enhance research outcomes. Learn about sampling strategies and sample sizes in research methodology.

allyson
Download Presentation

Understanding Study Designs: Experimental vs. Quasi-experimental

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 6: Study Designs Experimental vs. Quasi-experimental (Experimental consist of random assignment to treatment and control conditions) Two conflicting forces: 1. Sufficient rigor and specificity to make firm conclusions 2. Limitations of money, time, cooperation, and protection of human subjects limits the design options.

  2. Table 6-1. Conditions to be met to show causality

  3. Table 6-2. Types of validity

  4. Cross-sectional v. panel studies • Cross-sectional (or independent) data refer to data collected at one point in time • Panel data refer to data collected at multiple points in time among the same people (or units) • Technical, longitudinal data may be either cross-sectional or panel. • Cohort is a single panel.

  5. Cross-sectional vs. Panel Studies • Typically, cross-sectional studies are conducted when: 1. Generalizability is important 2. Theory being tested is new 3. Population is mobile or otherwise hard to contact • Panel surveys are useful when: 1. Confident that the intervention will have some effect and the researcher is interested in the process of the effects 2. Need more exact information on who did what 3. Can use with smaller sample sizes because variance estimator is more efficient.

  6. Table 6-3. Trade-offs between panel and cross-sectional design

  7. 4 Types of Validity • Study validity • Measurement validity • Statistical validity • Program validity

  8. External: Generalizability of research results Internal: Conclusions made are accurate External vs. Internal Validity

  9. Threats to Validity History - the occurrence of uncontrollable factors during the study Maturation - effect of time that cannot be controlled Testing - the increased ability of the subjects due to learning that occurred from taking of the test Instrumentation - the effect of data recording procedures which subjects may react to differently, and the ability of the instruments to measure what they purport to measure Statistical Regression- changes can occur by virtue of subjects starting out in extreme positions will be attributed erroneously to the effects of the experimental stimulus Sensitization- the interaction of the test and intervention: taking the test sensitizes the subject to topic of the intervention

  10. Threats to Validity (cont) • Selection biases - groups must be comparable • Experimental Mortality - subjects will drop out of experiments, thereby affecting research results • Causal time-order - ambiguity about time-order of the stimulus and dependent variables arise--dependent variable actually causes effect • Diffusion or imitation of treatments - if experimental and control subjects can communicate, some experimental stimulus could be passed on to control group; control group becomes affected by stimulus and therefore is not real control group

  11. Threats to Validity (cont) • Compensation - experimental group receiving something considered valuable, therefore control group might be given compensation, thereby no longer true control group • Compensatory rivalry - control subjects, deprived of stimulus,might work harder to compensate • Demoralization - feelings of deprivation might cause control group to give up

  12. Table 6-4. Study Designs O, observation, such as a survey; X, intervention/program; --, no observation or intervention

  13. Table 6-5. Hypothetical Study Outcome Scores (panel study)

  14. Description of Observation Groups

  15. Program and Validity Threat Estimates for the Hypothetical Panel Study

  16. Selecting a Study Design 1. Ethical issues - Study design should not contain any unethical procedures. 2. Real-world considerations (Practical and administrative issues) 3. Technical issues - 1. Random assignment to conditions 2. Finding comparison groups 3. Time series design 4. Collect baseline information 5. Multivariate techniques 6. Threats to validity

  17. Good Research Designs: 1. Are ethical - provide best quality of services 2. Provide the most reliable and valid data 3. Measures whatever happens in the field 4. Avoids making false conclusions

  18. Three Multiples 1. Multiple data sources a. provides reliability check b. provides additional insight into the process c. opportunity for qualitative information 2. Multiple measurements over time 3. Multiple replications a. generalizability b. controls for unanticipated events

  19. Chapter 7: Sampling & sample size • Sampling: What type of sample to select • Sample size & power: How many units are needed AND • If I have N units, is that sufficient? • Sample type and size will dramatically influence study costs and procedures.

  20. Probability Sampling 1. Simple random sampling - EPSEM 2. Systematic 3. Stratified & Cluster 4. Census 5. Multistage 6. Probability proportional to size (PPS)

  21. Non-Probability Sampling 1. Convenience 2. Purposive - planned selection of specific cases. a. Quota sampling - numbers of elements that fit certain criteria are included in the sample. b. Key informants - Non-probability, purposive selection of respondents who fit certain conditions.

  22. Sample Size Determination 1. Resources 2. Plan of analysis - the sample must be large enough to: a. Produce uni- and bivariate crosstabs. b. Provide for accurate estimates of proportions.

  23. Four Methods for Sample Size Calculation • Tabulation • Proportions • Differences • Power

  24. 1. Tabulation Method • At least 50 cases for each category of key independent variables • At least 5 cases in any cell of a cross tabulation • Not often used because in practice distributions are unknown

  25. 2. Proportions Method(Population greater than 10,000) n=z2pq/d2 • p= proportion (estimate: default is .5) • d=degree of accuracy (.05 or .01) • z=confidence interval (usually 95% level) For example, n= (1.96)2(.50)(.50) / (.05)2 = 384 Or more conveniently: n = 4(.50)(.50)/ .052 = 400 And still more conveniently: n = 1/d2

  26. Figure 7-1: Schematic of Multistage Sample Selection. Counties (Cluster) Blocks (Cluster) Households (Primary Sampling Unit) B A

  27. Correction Factors • Design effect - correction for clustering of sample. Some communities have more variation than others so you compute the ratio of variances to get design effect parameter • Sampling significant fraction of the population. If it exceeds 5% then use correction n/(1+n/N)

  28. 3. Differences between Proportions • Minimum difference you wish to be significant • n’ = 2z2pq/(d’)2 For example, assume .4 proportion and expect that a .1 difference is sig. n’ = 2(1.96)2(.40)(.60)/(.10)2 = 184 Thus you would need 200 subjects in the control and 200 in the treatment group

  29. Power Analysis • Sample size is determined by the power of the test (usually set to 80%): the precision (95% CIs); the critical value and direction (5%) ; and the effect size (meaningful impact level): n=2(z + z)2 p q / d2

  30. Power • Critical value, Alpha, Type 1 • Sample size • Effect size • Power, Beta, Type 2 error

  31. Understanding Hypothesis Testing

  32. Chapter 8: Data Collection Survey Question Design - Simplicity 1. Simple language - Use simple terms and sentence structure 2. Short questions 3. Avoid double-barreled questions 4. Use common concepts 5. Present manageable tasks - Don’t expect the respondent to keep in mind numerous categories. 6. Shared definitions - For example, when you use the term family what does it mean. Include all definitions and instructions in questionnaires. 7. Questionnaire construction easy-> specific -> demos

  33. Rules for Recall 1. Bounded 2. Reference period (six months or less) 3. “Average” - In a typical day, how many hours do you spend watching TV? 4. Landmarks 5. Provide Cues

  34. Experimental Evidence 1. Specificity - specific questions are more accurate 2. Close-ended questions 3. Offer a “no opinion” or “neutral” option 4. Include a middle category, but don’t read it 5. Measure intensity whenever possible 6. Be sensitive to question order funnel from general to specific 7. Wording effects are real and powerful 8. Ask multiple questions on a topic

  35. Tools at Hand 1. Circulate drafts of your questionnaire 2. Use FGDs and interviews w/ target audience 3. Borrow freely from standard ?-naires (Standard questionnaires must still be pre-tested since (a) language constantly changes, and (b) questions affected by neighboring questions) 4. Conduct declared and undeclared pretests

  36. Pre-test, pre-test, pre-test!! 1. Variation: you want a 30/70 variation not a 5/95 2. Meaning: How does the respondent interpret the question. (Ask them: “What does this mean to you?”) 3. Task difficulty - Make questions simple 4. Interest & Attention: Vary the question responses and phrasing

  37. Question Types • Dichotomous • Scaled response (don’t read middle categ.) • Balanced/unbalanced scaled response • Fill in the blank • Rating scaled response • Ranking scaled response • Semantic differential

  38. Box 8-1: Question Examples Dichotomous (binary choice) In the past 6 months, have you seen any commercial on TV related to illegal substance abuse? 0.  No 1.  Yes SCALED RESPONSE, BALANCED (LIKERT) How much did you like that TV commercial that you saw regarding substance abuse? Did you like it very much, somewhat like it, somewhat dislike it, or dislike it very much? •  Very much liked it 2.  Somewhat dislike it 3.  Somewhat liked it 4.  Very much disliked it SCALED RESPONSE, UNBALANCED In the past 6 months, how often have you seen commercials on TV related to illegal substance abuse? •  Never 2.  Sometimes 3.  Often 4.  All the time

  39. Box 8-1: Question Examples (cont.) SCALED RESPONSE, RATING How entertaining did you think each of the TV commercials was regarding substance abuse? Was [read name of commercial] very, somewhat, or not at all entertaining? [repeat for each commercial]

  40. Box 8-1: Question Examples (cont.) SCALED RESPONSE, RANKING Please rank the six TV commercials on substance abuse according to how entertaining each was. Which commercial was most entertaining? What second most? Third most? Fourth? Fifth? Least entertaining? [place the rank order next to each commercial.] Commercial A ____ Commercial B ____ Commercial C ____ Commercial D ____ Commercial E ____ Commercial F ____

  41. Box 8-1: Question Examples (cont.) SCALED RESPONSE, SEMANTIC DIFFERENTIAL For each commercial, I will read two words, and on the scale of 1 to 5, please indicate which world most closely resembles your reaction to the commercial. If the first world reflects your reaction, then respond with a 1, and if the second world reflects your reaction, respond with a 5, and if you feel in between, please provide the number that corresponds to your feelings CIRCLE THE NUMBER ON THE SCALE THAT MOST CLOSELY RESEMBLES YOUR REACTION TO [read for each commercial]

  42. Box 8-1: Question Examples (cont.) FILL IN THE BLANK In the past 6 months, how many different TV commercials related to illegal substance abuse have you seen? _____ (number of unique commercials seen)

  43. Pre-testing the Questionnaire 1. Flow: Does the survey flow 2. Question order: Do a question logically come after another 3. Skip patterns: Keep to a minimum and make sure you don’t lose anyone 4. Length: At least less than one hour, probably shorter (Phone < face-to-face) (Volunteer < Paid) (Commercial < Pro-social)

  44. Questionnaire Development 1. Close-ended questions (double cost) probe meaning in open-ended follow-ups 2. Length should be less than twice final length 3. Respondents should resemble target population 4. Pretest 25-75

  45. Survey Types Three main types: • Phone (30-50%) --> 70-90% • Mail (30-50%) --> 70-90% • In-person Degree of interaction Cost Which is best for what

  46. CATI/CASI • CATI advantageous in that skip patterns are responses are built in • Interviewer doesn’t need to handle paper • CASI adds flexibility and anonymity • CASI can include graphics and audiovisual aids; data are entered directly into the database reducing errors

  47. Response Rates Affected by: 1) relevance of study 2) reputation of surveyor 3) quality of the survey 4) persistence of researchers 5) incentives ($1/$5 in envelop) 6) first class postage 7) pre-notification and cover letters Estimates • Mail general population HH: (30-50%) --> 70-90% • Mail professional lists: (40-60%) --> 70-90% • Phone Surveys: (40-60%) --> 70-90% • In-person: 70%

  48. TDM • Specified importance of pre-notification letters, follow-up postcards, follow-up surveys, follow-up phone calls • Importance of mailing package and attractiveness of survey, incentives • Improved RRs dramatically make data more accurate

  49. Data Management • Dataset Formats - Spreadsheets, Database Programs, Statistical Programs, Editors • DBMSCOPY – used to copy datasets from one program to another

  50. Dataset Terminology Usually: 1. Cases=observations=records=respondents=rows 2. Variables=questions=columns (usually) 3. Variables have labels; values have labels; and values should be stored as numeric if they are numbers and strings if characters.

More Related