Immigrant respondents and quality in population surveys:

1. Immigrant respondents and quality in population surveys: sampling, non-response & questionnaire design Q2014, Vienna, 2.6.2014 Liisa Larja, Ada Kotilainen

2. Content of the presentation • Survey design • Fieldwork • Questionnaire design • Conclusions

3. Problem 1: immigrantsareunderrepresented in the general populationsurveys

4. Problem 2: Lowresponserateamongimmigrantsdecreases the quality of general populationsurveys

5. Problem 3: Disaggregating results according to immigrant status is not possible when there are too few immigrants in the general population survey samples • No results: problems go unnoticed. Separate studies: comparison? • EU immigrant integration indicators derive from Labour Force Survey (LFS), Survey on Income and Living Conditions (SILC) and PISA • LFS ad hoc module 2014 on Labour market situation of immigrants and their descendants  in the Finnish LFS the number or immigrants would be very low (~900). • Oversampling – expensive • Pooling of data from several years – timeliness and quality problems, not possible for LFS ad hoc module 2014

6. Solution: Survey design where the data for 3 surveys is collected within 1 interview

7. The ”UTH”-study (Survey on work and well-being among persons of foreign origin) • N = 5 400 individuals of ”foreign origin” (= both parents foreign-born)  all languages, all areas, citizens, foreigners, 2nd generation…. • Sampling from population register. At least 6 % overcoverage: moved out from Finland • Computer assisted personal interviews (CAPI). Normally done in respondents home, but 50 % of the respondents wished for some other place. • Standardized interview technique

8. 2. Field work • For the first 7 reference weeks the response rates have been over 70 % , which is slightly better than in corresponding general population surveys. • Translating questionnaires and programming them into Blaise also “monolingual” interviewers were able to use language versions • Translating cover letters and info sheets on data secrecy • Recruiting 9 multilingual interviewers (through NGO’s) in addition to 120 permanent interviewers

9. …fieldwork • Training, motivating & encouraging interview staff (sensitive questions, interpretation, motivating different types of respondents) • Investing in searching for missing contact information (different registers, home visits & notes, neighbours, googling, social media, e-mail, SMS). Average time spent per sampling unit 3,5 hours (interview 0,7 h). • Non-response due to same reasons as for general populations: non-contacts, no time, no interest.

10. 3. Questionnairetesting: an iterativeprocess • 21 cognitive interviews (incl. one proxy). Focus on new questions. How do the respondents interpret the questions, how do they choose their response, do they leave something unsaid? • 6 pilot interviews (incl. one ad hoc translation). Testing the entire questionnaire. Length and flow of the interview, spontaneous reactions to the questions. • 21 pilot interviews. Testing of languages versions, testing of distant translation by phone and video (+ length and spontaneous reactions)

11. Questionnaire design: Comprehension problems • Linguistic problems (difficult words and sentence structures, Likert scales)  incorrect interpretation of the questions  reformulation of the questions • Many misinterpretations emerged only in the cognitive interviews. • Standardized scales  “easy Finnish” versions + help-text, to decrease need to deviate from standardized interview technique • Unfamiliarity with the interaction pattern of structured interview  The length of the questionnaire was cut by 40 %  Investing in language versions.

12. Questionnaire design: translations • Finnish, Swedish and English + 9 additional languages.  68 % had language version in their mother tongue • The number of language versions was prioritized over quality of translation: • Professional translation, but no back-translation, committee translation etc. • Interviewers proof-read the questionnaires and only obvious mistakes were corrected. • With only a few (rigorously tested) language versions there would be even more misunderstanding due to broken Finnish/English as compared to linguistic differences due to differences in translations.

13. Questionnairetesting: trust • Questions on reasons for migration, language skills, cultural identity, labour market obstacles, health problems, experiences of violence, pregnancies etc. were found sensitive, especially with proxy respondent or when a friend was assisting with translation  routing was changed to skip some of these questions for proxy respondents and when there were other persons assisting in the interview  response cards were added to allow responding without speaking • The respondents expressed concerns on whether the information they give may have an effect on decisions on their social benefits etc.  info sheet on data secrecy • The respondents admitted, that the imagined opinion of the interviewer influenced option selection  interviewer training on sensitivity

14. Conclusions • Oversampling immigrants can be done cost-effectively by collecting data for multiple surveys during the same interview • High response rates can be achieved by investing in quality in field work (language version, working time in contacting) • Questionnaire testing is essential and it is best done iteratively • Finding solutions to the problems of comprehension, and trust is essential as they affect response behaviour and produce bias in the results.