Field Research Methods

1. Field Research Methods Lecture 3: Special Topics in Survey Design Edmund Malesky, Ph.D., UCSD

2. Outline of Today’s Lecture • Non-Response Bias • Missing Data • The Anchoring Problem • Differential Item Functioning. • Survey Experiments

3. 1. Non Response Bias (A Short Illustration) Click on the picture to play.

4. 1. Non-Response Bias • Single Rule of Probability Sampling: Each individual in the population has the same probability of being selected. • If some factor in your survey design/sampling reduces responses, it will bias your estimates. • The problem is figuring out what the direction and impact of the bias are. These are usually determined by the sampling approach and research question.

5. Non-Response Bias(Calculating the response rate) • Should be a pretty straightforward exercise: • If interviewing procedure involves an initial screening to identify vacant addresses, non-working telephone numbers, etc., these can be subtracted from the denominator. • But if screening call identifies members of the research population who refuse to participate in future work, these must be counted in the denominator. • If your research strategy involves replacement (refusals are replaced with new respondents to reach a target sample), these contacts must be counted. • In developing country research, adjustments to response rates sometimes must be made. (i.e. .75*response rate), if 25% of the population do not have a phone. • If data cleaning process rejects response, these must be subtracted from the numerator. • Note: The official American Association of Public Opinion Research (AAPOR) has a more rigorous definition of response rate. Let’s look.

6. Non-Response Bias(Ideal Response Rates) • Office of Management and Budget (OMB) requires a response rate in excess of 75% for research done under contract to the federal government. • The National Standard for Education Statistics requires a non-response bias analysis for anything less than 85%. • For mail-out surveys, it is not uncommon to find response rates of 50% or less. Less than 20% is pretty much useless. • In developing countries that are unfamiliar with surveys, it is reasonable to expect lower response rates. • This is especially true if the survey asks sensitive questions which are new to the respondents.

8. Potential Variance around Mean Response Fowler, p. 45

9. 1. Non-Response Bias (A Review of Measurement Error) • Remember, every causal research specification has two major components. • The systematic portion. The variance in y, which we can describe with our measured variables. • The stochastic portion. This is the random error.

10. 1. Non-Response Bias (4 Types Measurement Error) • Systematic Measurement Errors • Constant Bias • Variable Bias • Stochastic Measurement Errors • 3) Errors in the Dependent Variable • 4) Errors in the Independent Variables

11. Systematic but Constant Error Non-response bias constant across independent variable. Non-Response bias is constant across independent variable.

12. Systematic but Variable Bias Non-response bias varies across independent variable. Non-response bias varies across independent variable.

13. Stochastic Error in the Dependent Variable Non-response leads to noisy data. Inferences are less certain.

14. Stochastic Error in the Independent Variable Non-response leads to incorrect estimation of independent variable – stretching estimates.

15. 1. Non-Response Bias • All errors can pose problems for inference. • Systematic, constant variance can lead us to under/over-estimate descriptive statistics. • Stochastic measurement errors bias findings toward 0. • But the most problematic is systematic variable bias. • Researchers must take caution to make sure that non-response rates are not correlated with some variable of interest or will not affect in important variable of interest. • Common non-response biases are toward: • More educated populations • Wealthier populations • More aggravated populations (i.e. Prof. Evaluations) • More rural populations.

16. Tips to Avoid Non-Response Bias • General • Well-written and designed survey • Limited number of questions and wasted time • Accurately present goals of research. • Clear statement of respondent confidentiality. • Door-to-Door • Pleasant introductory phone-call, letter telling respondent that they have been selected. • If simply dropping by make sure interviewers are well-dressed, professional, and carrying a opening letter. • Arrange interviews for convenient times and places. • Do not appear rushed. • Gift to respondent in advance of interview.

17. Tips to Avoid Non-Response Bias • Telephone • Numerous phone calls • Call when respondent is likely to be home (hire interviewers with flexible schedules). • Send introductory letter if possible. • Mail-out/Internet • Anything that makes survey seem more professional (layout, colors, envelope). • Follow-up phone calls and cards to non-respondents. • Avoid large open-ended questions. • Pre-payment - Some companies have increased response rates tremendously by simply sticking $1 in the envelope. • Gift – In Vietnam, companies can win VCCI publications

18. Post-Hoc Non-Response Corrections • Re-weighting based on population percentages of under-sample populations. • Follow-up survey of non-respondents.

19. 2. Missing Data • What happens when a respondent agreed to the survey, but refused to answer particular questions? • Or checked the don’t know option when all indications are that they should have known the answer? • How do we address this research problem?

20. 2. Avoiding Missing Data in the Survey • Address sensitive questions as carefully as possible. • Allow the respondent to answer about a general group (projecting away from the individually) • Let the respondent know you are sympathetic to the circumstances and aware of the problem. • Avoid too many memory-intensive questions or calculations.

21. 2. Addressing Missing Data after Collection • 3 Types of Missing Data (King et al 2001) • Missing Completely at RandomBasically, a respondent answered questions in the survey based on coin-flips) (Very Rare): • Missing at Random: Missing data can be predicted with other values in the dataset. (i.e. Large, urban firms are less likely to answer corruption questions). • Non-Ignorable: Missing data can only be predicted with unobservable information not in the dataset.

22. Problematic solutions to Missing Data • Listwise Deletion: Dropping observations with missing information (very common). • Leads to inefficient (larger variances) results for MCAR and biased & inefficient results for MAR. Respondents are choosing not to respond for a reason. • King shows it is not any better than the omitted variable bias which results from dropping the variable. • Best Guess Imputation: Basically, filling-in data. • Depends on the guesser!! • Imputing a 0 and then adding an dummy variable to control for the imputed value. • Leads to biased results. • Mean substitution • Attenuates estimated relationships.

23. King and Schaefer Solution: Multiple Imputation • If MAR or MCAR apply, King argues that we can predict the missing data based on observable already in the set. • Using a simulation procedure (similar to Clarify) we predict the missing data many times, storing the predictions in multiple datasets. Observed values are the same across the sets, but predicted values for an individual observation will vary. • When we run our analysis, we use all the datasets simultaneously (sometimes as many as 10). • The final result includes the average effect across the 10 datasets. • King’s program Amelia performs these procedures automatically within STATA. • Monte Carlo simulations prove that the procedure is less biased and more efficient than all other methods.

25. 3. The Anchoring Problem(A Short Illustration) Click on the picture to play.

26. 2. The Anchoring Problem(Two Big Problems in Survey Research) • How to measure “big” concepts we can define only by example • E.g., freedom, political efficacy, pornography, health, etc. • The usual advice: You do not have a methodological problem. • Get a theory and it will produce a more concrete question. • The result of more concreteness: more reliability, no more validity • 2. How to ensure interpersonal and cross-population comparability • Chinese report having more political efficacy than Americans • The most common measure of health — “How healthy are you? (Excellent, Good, Fair, Poor)” — often correlates negatively with actual health • Amartya Sen (2002): “The state of Kerala has the highest levels of literacy. . . and longevity. . . in India. But it also has, by a very wide margin, the highest rate of reported morbidity among all Indian states. . . . At the other extreme, states with low longevity, with woeful medical and educational facilities, such as Bihar, have the lowest rates of reported morbidity in India.” • Brady (1985): “Individuals understand the ‘same’ questions in vastly different ways.”

27. The Anchoring Dilemma “Is your province creative and clever about working within central law to solve problems for firms like yours?”

28. Chinese have more political efficacy than Mexicans?

29. Previous Work on Anchoring • Lingo : Differential Item Functioning (DIF) • Approach: Find anchors to attach response categories to a standard metric. • Previous Attempts • Clearly label end-points (hawks, doves..) • Self-Anchoring (Label end-points with most liberal and conservative friends) • Designated anchors (Questions experts say have no DIF). • “Factor Analysis” Average all other questions to test each member. • Identify questions without DIF (raw numbers) or collapse categories to reduce DIF • Find over-estimations on other questions by comparing to an objective ranking and “correct” questions with DIF by re-weighting. • Multi-dimension scaling (i.e. Poole and Rosenthal Scores). • Aldrich and McKelvely (Estimate true candidate scores, based on other candidates ranking).

30. An Anchoring Vignette Example

31. Non-Parametric Solution:Which has more political efficacy?

32. The Math Behind the Simple Anchoring Correction

34. The Non-Parametric Solution Applied to China and Mexico

35. Click on anchor to view vignettes from different fields

36. Key Assumptions of the Anchoring Vignette Approach? 1. Response Consistency: Each respondent uses the self-assessment and vignette categories in approximately the same way across questions.(DIF occurs across respondents, not across questions for any one respondent.)2. Vignette Equivalence: (a) The actual level for any vignette is understood in the same way (on average) for all respondents. (b) The quantity being estimated exists. (c) The scale being tapped is perceived as unidimensional. 3. In other words: we allow response-category DIF but assume stem question equivalence.

37. Practical Issues in Anchoring Vignettes • They take up a lot of time and space on a survey. • Solution: Do not use them on all respondents; only a representative sample. • Elites hate them! • Be aware of double-barreled vignettes. • Pay attention to name selection. Names may be loaded with gender and socio-economic meaning that could unintentionally introduce framing effects. • Randomize order that respondents hear the vignette to ensure that the order does not drive results. • Train enumerators carefully. This is a very very diffcult question to administer.

39. Show Card

42. 4. Survey Experiments • As we have seen, cross-sectional “single-shot” surveys are fraught with design problems that limit their utility. These include: • Selection bias • Spurious correlation • Correlated measurement errors • Mutual Causation (Endogeneity)

43. 4. Survey Experiments • If properly executed, the survey experiment may get the researcher around many of these problems. • Experiment uses the power of large numbers and randomization to verbally get around these issues. • One question supplies a treatment, which is randomly administered, followed by a follow-up question on the effects. • Essentially, survey experiments take advantage of the “framing effects” in surveys.

44. Frye Example 1

47. Frye Example 2

48. My Experiment on First Years

49. Results Percentage of respondents more likely to vote for governor after treatment. Percentage of respondents with collapsed treatment

50. Getting Survey Experiments Right • The intervention must be reasonable and believable. • Avoid highly obtrusive treatments (low intensity) • Manipulate information not emotion! • Manipulate the frame or the choice-set. Do not manipulate (through arousal or deception) the characteristics of the chooser. • Remember to include a control group. • If you have more than one experiment in a survey, be careful to avoid contamination of subsequent surveys.