Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn

1. Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn ONS Social Survey Division

2. Overview • Background and context setting • The pilot study; our key objectives and design • Findings • Next steps

3. Pilot Key Objectives – ‘Dipping our Toes’ • Need to examine web collection • Investigate unit and item response rates • Investigate respondent characteristics • Evaluate the hardware/software capabilities Examine the effect of offering an incentive • Design and administration of web surveys • Recommendations for further work • Deliver to a short timescale (by December 2008)

4. Sample Design and Respondent Selection – Web Pilot • Simple random sample of 2,000 households from English Postal Address File • Advance letter • Half offered a £10 Amazon gift certificate to take part online • Letter asked adult with most recent birthday to complete the survey • Unique username and password provided to access survey from ONS website • Follow-up letter

5. Sample Design and Respondent Selection- Opinions Survey (OPN) • OPN is multi-stage cluster design • Respondent selection via KISH grid • November OPN sampled 2,010 households • Book of postal stamps sent with advance letter

6. Based upon ONS Opinions Survey (OPN) Used question blocks equivalent to a ‘normal’ month of OPN Questions reviewed for suitability for web Some modifications necessary Questionnaire tested internally Questionnaire Design (1)

7. Questionnaire Design (2) • Design and layout reviewed for web survey* • Page Design • Next/Back Button • Progress Bar * See Dillman, D.A (2007) Mail and internet surveys - the tailored design method (2nd edition) J Wiley & Sons, New Jersey

8. Questionnaire Design (2)

9. Unit Response Internet pilot • 364 fully complete responses received (18%) • 32 partial responses (2%) • Little difference between incentive (18%) and non- incentive (19%) group • But, Full sample= 2000 10% unknown eligibility and only 61% have internet access at home (LCF) Available sample = 1098 New Response = 33% complete responses November OPN • 59% overall response (with known eligibility)

10. Metrics • 80% of respondents completed web survey in under 25 minutes • 75% completed 8am-6pm • Most popular hours to begin survey were 4pm-5pm (10%) and 7pm-8pm (11%) • 25% completed on weekend • Wednesdays and Fridays most popular • OPN face to face takes c.45mins (excl. admin time)

11. Results (1) • Weighted results; OPN routinely weighted but pilot required some methodological work. OPN was then re-weighted to same population totals. • 2-tailed t test applied to differences • Some clear differences in profile of internet pilot respondents compared to OPN and key question responses

12. Results (2) • The pilot respondents were more likely to be: • Aged 25-44 and 55-64 • Married and living with partner • White • Better educated (with degree level qualification) • Managers or supervisors • In good health

13. Results (3) • In terms of key question responses pilot respondents were less likely to: • Smoke • Have a disability • Think charities played an important role in society • Think HM Revenue & Customs treated them fairly • Pilot respondents were more likely to: • Eat healthily (5 portions of fruit or vegetables a day)

14. Next Steps • Limitations of this pilot widely accepted • Pilot has demonstrated viability of conducting a major ONS social survey online and receiving a significant, if minority, response • Pilot has highlighted logistical and technological issues • Pilot has highlighted risk of significant bias • Identified areas for further work via more sophisticated pilot designs • Plan to run a further pilot on OPN in 2010 and pilot on LFS early in 2011

15. References • Dillman, D.A. 2007. Mail and Internet Surveys – The Tailored Design Method (2nd ed). J Wiley and Sons, New Jersey • Flatley, J. 2001. The Internet as a Mode of Data Collection in Government Social Surveys: Issues and Investigation Social Survey Methodology Bulletin, ONS 49(7) • Fricker, R and Schonlau, M. 2002. Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Field Methods 14(4) • Solomon, D. 2001. Conducting Web-Based Surveys. Practical Assessment, Research and Evaluation 7(19)

16. Internet and the UK LFSDean FletcherUK ONS

17. Internet and the UK LFS • Current Modes: • Wave 1 Face to Face Wave 2-5 TU or ftf • Response on UK LFS all time low 65% wave 1, 57% overall. • Achieved sample size decreasing or much more costly to obtain • Efficiency savings face to face costs 3 times TU • Pilot shows some groups with current high non-response will take part in the internet. • TU costs 3 times Internet.

18. Internet and the UK LFS • Pilot showed technology exists. • 4 main Problems: • Collecting close to the Reference week • Problem of integrating the internet with the other modes to prevent bias • Mode effects • Collecting from Households not persons

19. Registered for internet Respond to internet FtF W1 TU W2-5 Internet and the UK LFS • Analysed other approaches but derived duel sample approach after considering results of the pilot Initial Sample Face to Face W1 TU W2-5

20. Internet and the UK LFS • Currently reviewing questionnaire for internet suitability based on pilot results and others research to minimise mode effects. • Household collection seems feasible • Use Confirmit to test ideas • Use BLAISE IS? To run pilot in 2011 in a few NUTS2 areas • If successful run full design by 2012 • Modify collection by 2013

21. Internet and the UK LFS • Issues • Internet pushes up refusals • Unequal response • Changes to internet over time

22. Internet and the UK LFS • Also monitoring wave 2-5 work by other NSi’s to implement if shown to be successful • But current systems do not allow internet, TU and ftf integration. • Cost savings and non-response mainly achieved for UK at Wave 1.

23. Internet and the UK LFS • So far we have had: • OPN Pilot • Input at International Workshop • Prof. Mick Couper’s work and training • Literature review • But interested to have design scrutinised for issues.

24. Internet and the UK LFS • Concluding remark • UK Can’t afford not to implement an internet collection mechanism if possible in the next 2 years.