THE ART AND SCIENCE OF FORMULATING SURVEY QUESTIONS

1. THE ART AND SCIENCE OF FORMULATING SURVEY QUESTIONS

2. Principal Sources of Error in Survey Results Sampling error The sample of interviewees is not representative of the universe from which it is drawn; it is skewed in various ways. (“Sample of the whole” eliminates bias) Respondent error (can be reduced by skillful questionnaire construction) Response is not appropriate/relevant to the question, due to misunderstanding (= poorly worded question?) Response is deliberately misleading (= lying) Faulty recall (retrospective questions) Social desirability response set: Response is determined by what interviewee believes is socially acceptable behavior or attitude Acquiescence response set: Respondent agrees with whatever statement is presented to him/her, for fear of offending interviewer Interviewer error Question not asked (skipped), question was asked of inappropriate respondent, response categories were read (directing respondent), response was inaccurately recorded, etc. Data processing error “GIGO”: garbage in, garbage out. Errors in transferring data from questionnaire into data base. (Electronic data capture eliminates such errors)

3. From research objective to survey question: An example • Objective: Document and explain change in rate of settlement in U.S. (vs. circular migration). • Explanatory hypotheses: Such changes may be a function of:-- Continued tightening of border controls, increasing coyote fees, indebtedness, and physical risk-- Family reunification migration | (fewer relatives to maintain in hometown)-- Maturation of social networks-- Less seasonal work in U.S., more employment in “permanent” year-round jobs-- “Anchoring” investments in U.S. (home ownership, small businesses)-- Life events (marriage, children reaching school age, etc.)-- Changing gender roles in migrant families (greater female labor force participation, etc.)-- Decisions to naturalize (gain U.S. citizenship), for “defensive” purposes, employment eligibility, etc. • Empirical tests/indicators? • Survey questions?

4. Filter questions: Types of Survey Questions • ¿Es Ud. de Tlacuitapa? (si NO es de Tlacuitapa, agradecerle a la persona y busca otra persona) • ¿Cuántos años tiene Usted? (si tiene menos de 15 años o más de 65 años, pedir hablar con otra persona) The answers to these questions are not marked on the cuestionario.

5. Standardized Question, Closed Response

6. Non-standardized Questions, Closed Response

7. Standardized Question, Open Response(avoid as much as possible) (Si no usó coyote:) ¿Cómo logró cruzar la frontera, sin usar coyote? (Anotar respuesta completa) ____________________________________________________________________ ____________________________________________________________________ ________________________________________________________________________________________________________________________________________

8. Open Question, Open Response • Ya no es encuesta. It’s a qualitative, life-history interview. • Questions soliciting answers that will involve a lot of complexity, nuance, and detail should be in a semi-structured interview schedule. Qualitative interviews are also generally better for getting at emotion, meaning, and narrative. • E.g.: ¿Qué es los que más/menos le gusta de los EE.UU.?

9. Questions with a numeric response: Different Types of Closed Responses

10. A poorly constructed question: A better question with a simple numerical response: ¿Cuántas veces a la semana fuma Ud.? ________ (Specify a time frame and a frequency of behavior. Answer may be “0.”)

11. Questions with a dichotomous yes/no response:

12. Questions with scaled responses:

13. “Thermometer” scaled response • (MOSTRAR TARJETA ) Le voy a preguntar su opinión acerca de algunas personas e instituciones. En una escala de 0 a 10, donde cero significa que su opinión es muy mala y el 10 que su opinión es muy buena, ¿cuál es su opinión acerca de….? (LEER Y ROTAR) • Seguro Social • Programa “Oportunidades” • c. El Papa Benedicto Dieciséis • d. Barack Obama, presidente de EstadosUnidos • e. Enrique Pena Nieto, presidente de Mexico

15. Dichotomous vs Scaled Responses: 55. Actualmente,si uno no tiene papeles, ¿es muy peligroso cruzar la frontera? Si 1 No 2 N/A 8888 NS/NQR 9999 Better: 55. Actualmente,¿qué tan peligroso es cruzar la frontera, sin papeles? Muy peligroso 1 Algo peligroso 2 Poco peligroso 3 Nada peligroso 4 N/A 8888 NS/NQR 9999

16. Questions with unscaledcategorical responses:

17. Comparative opinion question (paired alternatives): 91. Algunas personas dicen que los jovenes nacidos en Tlacuitapa pueden progresar en la vida sin salir del pueblo. Otras personas dicen que para superarse, los jovenes nacidos en Tlacuitapa tienen que salir. ¿Qué diria Ud.? puede superarse sin salir 1 tiene que salir 2 N/A 8888 NS/NQR 9999

18. Questions asking respondent for a rank ordering:

19. OTHER ISSUES IN SURVEY DESIGNDefine complex concepts precisely ¿Cuántosviajes a hecho a los EstadosUnidos, paratrabajar o vivir? Definición de ‘un viaje’: Un viajeesuna estancia en los EE.UU. quedurapor lo menos 1 mesy cuyomotivo principal estrabajar o vivir. Noincluyevacacionesniturismo. Si el entrevistadoregresa a Tlacuitapapormenos de 1 mes y luegoregresa a los EE.UU., consideramosqueesel mismoviajea los EE.UU.

20. Always specify the time frame for reporting 21. En los últimos 5 años, ¿dónde ha pasado más tiempo, en Tlacuitapao en los EEUU? San Miguel Tlacotepec 1 Otra localidad en México 2 EEUU 3 N/A 8888 NS/NQR 9999 22. En los últimos 5 años, ¿cuántas veces regresó Ud. a Tlacuitapa? Nº de veces: _________ N/A 8888 NS/NQR 9999

21. Avoid questions that don’t elicit sufficiently detailed responses 55. Actualmente,si uno no tiene papeles, ¿es muy peligroso cruzar la frontera? Si 1 No 2 N/A 8888 NS/NQR 9999 Better: 55. Actualmente,¿qué tan peligroso es cruzar la frontera, sin papeles? Muy peligroso 1 Algo peligroso 2 Poco peligroso 3 Nada peligroso 4 N/A 8888 NS/NQR 9999

22. Softening the question/reducing sensitivity Ahora quiero hacerle algunas preguntas sobre su último intento o éxito en cruzar la frontera. 30. ¿Lo hizo con papeles, o tuvo que entrar sin papeles? Con papeles legales 1   Con papeles prestados 2 Con papeles falsos o chuecos 3 Sin papeles 4   N/A 8888 NS/NQR 9999

23. Organization of questionnaire/sequence of items Sensitivity of items 0 Time elapsed in interview

24. The art (and a little of the science) of field interviewing

25. Interview each other to gain confidence

26. Get your bearings

27. Build rapport with the entire community

28. Try to convince Profe that you’re not just having a pachanga

29. Key issues/challenges Introducing yourselfHow to explain what you are doing to the potential interviewee. “How will I – or my community – benefit?”(Para que sirve todo esto?) Hints: • Have an honest “sales pitch.” • We are writing the “history” of the town and its migration. • Our UCSD group was here three years ago. • “Confident assertiveness” – Fowler • We’re not from the US or Mexican government. • Information is anonymous. • Participation is voluntary.

30. What are the benefits for Tlacuitapa? • Don’t promise anything we can’t deliver. • Community leaders are interested in high-quality data about the health of residents and migrants • Our research findings can be used to educate U.S. public and “funcionarios” who make U.S. immigration policy – para promover la reforma migratoria. • Smaller expressions of gratitude: passing out free books from previous Tlacuitapa studies (selectively!), offering to take family photos to migrants in the U.S., etc… • Benefits tend to be at the community-level for the town & binational society rather than for specific individuals.

31. Managing social distance issues: You are an “elite” college student: perceived as upper-class, highly educated, culturally different. How to make yourself less threatening? • Hints: • Allow the interviewee to ask you questions about yourself, to reduce social distance and establish trust. • Trade on your student status. You are there to learn. • Emphasize our team’s previous experience in Tunkas and southern California; common social ties. • We are an “equipo binacional.” • Draw on the gender/ ethnic/ national diversity of the team to manage difficult interview situations.

32. How to manage time during the interviewNeed to keep to a schedule, to complete interview in a reasonable amount of time, while maintaining the flexibility needed to build rapport and stimulate informative exchanges of information with interviewee.Hints: -- allow interviewee to tell you (some) of his own stories;-- use pauses, allowing enough time for a response (don’t interrupt silence); -- internalize the questionnaire (memorize it!) so you don’t waste time looking for the next relevant question)

33. Read the question the first time EXACTLY as it is written in the cuestionario. This is especially important for attitudinal questions. If the respondent doesn’t understand the question, you may explain it in your own words. Probing: How deep to probe, to get a response that’s more detailed, more complete, more reliable, more relevant to the question?Caution: Danger is that, with extensive probing, interviewee may “invent” an opinion that in reality he doesn’t have, to be courteous or to avoid the shame of not having an opinion.

34. Detecting and correcting errors of inconsistency in responses Hints: • “Active listening” is important. • Pay very close attention to responses. Think about the response content. • Repeat the respondent’s own words to signal your understanding of him, and to set up follow-up question to clarify a contradiction or inconsistency.

35. Reducing respondent error (“reporting error”) • Interviewee doesn’t understand question, gives inappropriate response. • Interviewee gives incomplete answers (not telling the whole truth) • Interviewee gives deliberately misleading answers (untruthful responses) • Interviewee has recall problems (poorly recalled facts, e.g., in response to retrospective questions about migration & employment history) Hint: Prompt respondent with questions about major life events like birth dates of kids, marriage, to help establish a timeline for other events

36. Other sources of respondent error • Social desirability response set (bias): Interviewee tells you what he thinks YOU want him to say, or what is most socially acceptable in his community, in the U.S. • Absurdly high values: For example, 40-year-old man tells you that he has had 10 wives or has been to the U.S. 100+ times, to impress you, or to signal he’s not taking interview seriously. • Non-response: Interviewee refuses to answer certain questions at all (due to fear, shame, desire to conceal income, etc.)Hint: Try to convince respondent of the non-threatening nature of the study. Try to get answers for all questions, but if the respondent refuses, move on to other questions.

37. Still more challenges • How to record responses (note-taking) without interrupting flow of the interview? • How to assure confidentiality? • How to respond to interviewee requests, for money, transportation, introductions, information about “El Norte,” etc.? • What “body language” should you use to build rapport and elicit more useful responses?