What is Understanding Society?

1. Understanding Society - Innovations in Panel Data Collection and AnalysisMichaela Benzeval, Peter Lynn, Tarek Al BaghalISER, University of Essex7th ESRC Research Methods Festival5th July 2016

2. What is Understanding Society? • Understanding Society: the UK Household Longitudinal Study :longitudinal household panel surveyof all ages, designed to track all residents in households and analyse change at individual and household levelby examining • what people are experiencing: health, circumstances and key events in their lives as they happen • attitudes, identity and subjective well-being • how individuals and households respond to policies and key events • linked administrative record • Incorporates British Household Panel Survey (started 1991) • Innovation Panel: sample for methodological research and experiments • Funded by ESRC, and number GovDept, publicly available data • Part of family UK longitudinal and worldwide panel studies

3. Innovation in Understanding Society • Overall design • Innovation Panel: Testbed for experimentation- internal- external • Incorporating biomarkers and genetic information • Data linkage, new technologies and new data Today • introducing mixed mode data collection in a panel survey • improving measurement in a panel context • targeting the design of panel surveys

4. Design (1)

5. Design (2)

6. How can researchers get involved in the IP? • Annual “IP Competition” for content of the questionnaire: • Small-scale experiments/methodological evaluations • Use longitudinal aspect of the survey • Data collection free for successful proposers • “Associated Studies” • Data collection separate from IP interview, sub-sample • Using mixed methods, e.g. qualitative interviews • Data collection funded and undertaken by proposer • https://www.understandingsociety.ac.uk/about/innovation-panel

7. Accessing the data and findings Study documentation: • Protocols, fieldwork reports, questionnaires, variable level metadata • User guides, Quality Profile, technical reports • Working Papers and publications from methodological research See www.understandingsociety.ac.uk Data are available at the UK Data Service, University of Essex: • Wave 1 - 5 data, including biomarkers; wave 6 and immigrant & ethnic minority boost sample to be deposited November 2016 • Innovation Panel data waves 1 -7 (wave 8 to be deposited July 2016) • Linked education data, geographical identifiers available by special licence or via secure lab See UK Data Service website: http://ukdataservice.ac.uk/

8. Introducing mixed mode data collection in a panel surveyPeter LynnISER, University of Essex

9. Survey Modes • Face-to-face interviewing (“CAPI”) • Telephone interviewing (“CATI”) • Self-completion on paper • Self-completion online (“web survey”)

10. Survey Tasks • Contacting the sample member • Persuading the sample member to take part • Sample member answers the questions (“data collection”) Contacting and data collection need not necessarily use the same mode However, it is efficient (in terms of costs and logistics) if they do

11. Relative Merits of Modes • Probability/ease of contact: Mail, Email > CATI > CAPI • Probability/ease of co-operation:CAPI > CATI > Paper SC > Web • Quality of answers:Completeness/detail: CAPI > CATI > Paper/WebHonesty: Paper / Web > CATI > CAPIEffects of visual vs. aural presentationEffects of interviewer presence • Costs: Web > Paper > CATI > CAPI • Constraints: sampling frames

12. Understanding Society Context I Solely CAPI initially: • To get best possible initial response rates; • To get respondent commitment; • To collect complex and sensitive data But with intention to move towards including web: • To reduce data collection costs • To reflect respondent preferences/norms

13. Context II ESRC ITT for waves 6 – 8: “There is a critical expectation that mixed modes of data collection will be introduced to the main study by wave 7. The proposal will include a SWOT analysis of the proposed new modes, plans for transition (including with fieldwork collection agencies) and identify potential cost savings in the short, medium and long term.”

14. Context III ISER Response: “proposed design uses modes sequentially…. sample members are initially invited to complete the survey by web; non-respondents to the web survey are issued to face-to-face interviewing; in a final stage non-contacted or refusal cases are issued to telephone interviews. The conditions for implementing the mixed modes strategy on the main survey are 1) that the approach used does not significantly raise overall attrition compared with using face-to-face interviewing only; 2) that the approach used leads to significant cost savings at least in medium term, and 3) that it maintains data quality comparable with face-to-face interviewing. ”

15. Mixed Mode Experiments • UKHLS Innovation Panel has carried several experiments with mixed-mode data collection: • Wave 2 (2009): (CATI+CAPI) vs. CAPI • Wave 5 (2012): (Web+CAPI) vs. CAPI • Waves 6-8 (2013-15): (Web+CAPI+CATI) vs. (CAPI+Web) • (Waves 1, 3, 4 solely face-to-face) • The experiments were multiple purpose. Prime purposes were to study effects on non-response, attrition and costs.

16. IP2 Experiment: CATI • Wave 2 sample randomly allocated to 3 equal-size groups (n ≈ 900 per group): • Face-to-face only (control group); • “Early transfer” mixed mode (treatment 1): CATI, but switch to CAPI as soon as it is clear that an interviewer visit to the household will be necessary; • “Late transfer” mixed mode (treatment 2): CATI, and try to interview as many household members by phone before switching to CAPI:

17. 3 Key Research Questions What are the relative effects of the three mode protocols on: • Response rates and attrition? • Sample composition? • Survey costs?

18. Some findings from…. I

19. Mixed Modes and Attrition Base = respondents at wave 1

20. Mixed Modes and Attrition Base = respondents at wave 1

21. Mixed Modes and Attrition Base = respondents at wave 1

22. Early Transfer and Attrition Base = respondents at wave 1

23. Late Transfer and Attrition Base = respondents at wave 1

24. ResponseRate • “Recovery” entirely restricted to “Late transfer” treatment

25. Sample Composition Logistic regressions: • Around 10% of tested effects ‘significant’ at 0.05 level • Little more than would be expected by chance • Not systematic R-indicators: • Differences between protocols in non-response bias appear minimal

26. Survey Costs Wave 2: In MM sample, 52% of households required an interviewer visit • Significant cost saving compared to CAPI • The saving did not differ between the two MM protocols Waves 3, 4: • No continuing cost savings if revert to CAPI

27. CATI Conclusions I 1. Sequential MM (Tele>CAPI) damaged response rates, relative to CAPI-only • For some sample members, a telephone approach reduces propensity to co-operate with a subsequent face-to-face request • The mechanisms by which this happens are unclear 2. When subsequent waves are CAPI-only, the RR difference between groups erodes • Occasional implementation of a MM wave on a CAPI-panel may not do long-term damage to cumulative RR

28. CATI Conclusions II 3. The erosion of the RR differential occurred only with the telephone-intensive protocol • Specifically, those who responded at w2 with tele-light were more likely to drop out at w3 or w4 than those who responded with tele-intensive • The effect was restricted to households with 2+ adults • Perhaps caused via intra-hhd communication and the salience of a face-to-face interview. Perceived inequity? Acceptability? Perceived lack of persistence?

29. CATI Conclusions III 4. Few differences in non-response bias between groups • Differential effects of MM may operate broadly across subgroups 5. Considerable cost savings are possible with MM, compared to face-to-face • But these do not continue at subsequent waves if mode reverts to face-to-face • And savings decline over waves even if mode remains CATI+CAPI, as proportion of interviewer time spent travelling and contacting reduces

30. IP5 Experiment: Web Two components: • Original sample, for whom this was the 5th wave; • Refreshment sample, for whom this was the 2nd wave. Households randomly assigned to one of two treatment groups: • CAPI (one-third of each sample); • Mixed mode (two-thirds of each sample). Mixed modes treatment: • Letter with URL and ID. Sent also by email where available; • Hhd grid (first respondent only), hhd qre (first householder), individual qre • 3 email reminders at 3-day intervals; After 14 days, reminder letter + face-to-face visits began

31. 3 Key Research Questions What are the relative effects of the three mode protocols on: • Survey costs? • Unit response rates? • Item response rates?

32. Some findings from…. I

33. Costs: Households Responding Completely by Web Analysis restricted to £10 incentive group • One-fifth to one-quarter of households do not require an interviewer visit: considerable cost savings if sample is large

34. Households Responding Completely by Web are more likely to be: • With broadband access; • All adults in household use internet; • Not more than 2 adults in household; • In urban areas

35. Household Response Rates

36. Household Response Rates

37. Household Response Rates

38. Household Response Rates

39. Individual Response Rates

40. Individual Response Rates

41. Effects of Incentives

42. Effects of Incentives

43. Mixed Modes over Multiple Waves Response rates (based on all issued at wave 5). Odds ratios: Source: Wave 5: Calculations based on Jackle, A. et al (2015) Survey Research Methods; Waves 6-8: From Gaia, A. (2016) ‘The effect of a switch to a mixed-mode design on panel attrition: Evidence from an experiment of the Understanding Society Innovation Panel,’ paper presented at International Panel Survey Methods Workshop, Berlin, June

44. Mixed Modes over Multiple Waves Percentage of completed interviews completed by web: Analysis restricted to original and IP4 refreshment samples, as IP7 refreshment sample was treated web-first for the first time at IP9. Source: Own calculations

45. Item Non-Response P: P-values from Wald tests of means adjusted for sample design

46. IP6 Experiment: Reducing Item Non-Response Three treatments. Variation is in what appears on screen following an attempt to skip a question: • Control: question repeated with DK/PNS options now included; • Reactive motivational: question repeated, with motivational statement:“If possible, please provide an answer to this question, as this is one of the key questions in this study. Please be assured that the information you give us will be treated confidentially.” • Follow-up motivational: As control, but if any DK/PNS answers, these questions repeated at end of interview:“Thank you for taking part in “Understanding Society” this year. Earlier in the interview there were some questions that you did not answer. As you know, you do not have to answer any question you do not want to. However, X of these questions are important to researchers and we would be grateful if you could try your best to answer them.” Treatments applied only to 6 questions with high item NR risk

47. Some findings from…. I