1 / 31

Improved Questionnaire Design Yields Better Data: Experiences from the UK’s Annual Survey of Hours and Earnings

Improved Questionnaire Design Yields Better Data: Experiences from the UK’s Annual Survey of Hours and Earnings. Jacqui Jones, Pete Brodie, Sarah Williams & Jane Carter UK Office for National Statistics. Overview. Data Collection Methodology at ONS Methods standards Design standards

juliana
Download Presentation

Improved Questionnaire Design Yields Better Data: Experiences from the UK’s Annual Survey of Hours and Earnings

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improved Questionnaire Design Yields Better Data: Experiences from the UK’s Annual Survey of Hours and Earnings Jacqui Jones, Pete Brodie, Sarah Williams & Jane Carter UK Office for National Statistics

  2. Overview • Data Collection Methodology at ONS • Methods standards • Design standards • Case study: The Annual Survey of Hours and Earnings Slide 1

  3. Background to DCM at ONS • Mid 1990s 1st QTU set-up for Census • Then expanded to include social surveys and business surveys Slide 2

  4. Methods Standards • Aim was to develop a framework of processes to be used for all DCM question(naire) development work • Outcome was the Data Collection Methodology Improvement and Measurement Framework (DCM IMF) Slide 3

  5. The DCM IMF Identification of data requirements Expert Review Evaluation Review + + Stage 1 Redesign & Development Stage 2 Pre-field Testing Stage 3 Field Testing Stage 4 Field Implementation Stage 5 Post-implementation Evaluation Stage 6 Identify and Disseminate Best Practice Stage 7 Slide 4

  6. The DCM IMF Expert review Evaluation review Stage 1 Stage 2 Stage 3 Stage 4 Literature review Instrument development Expert review Qualitative testing Re-development Expert review Qualitative follow-up Quantitative analysis Slide 5

  7. Design Standards • Aim was to develop tested standards for the design of questions and questionnaires • Outcome is draft standards – with constraints – colour and cross boxes Slide 6

  8. Example of Design Standards

  9. Example of Design Standards

  10. Case Study: ASHE • ASHE collects data for earning statistics e.g. minimum wage • ASHE questionnaires are sent out to employers to complete on behalf of selected employees • 250,000 sample of employees Slide 9

  11. ASHE: Need for Change • National Statistics Quality Review • New data requirements e.g. • Pay for other reasons • Pension data • Existing data problems e.g. • Possible inclusion of pay elements in the wrong responses Slide 10

  12. Overview of the ‘Old’ Questionnaire • Very cramped 2 sided questionnaire which collected 26 data items • 4 sides of additional notes • Respondent coding Slide 11

  13. ‘Old’ Questionnaire Slide 12

  14. ‘Old’ Questionnaire Slide 14

  15. Overview of the ‘Old’ Questionnaire • Unclear data requirements • Inconsistent terminology • Double-barrelled questions Slide 15

  16. ‘Old’ Questionnaire

  17. ASHE Development Followed DCM IMF processes: • Focus group (8 large businesses) • Pre-field testing • 30 cognitive interviews (5 waves of testing) • Waves 1-3 concurrent interviewing • Waves 4-5 retrospective interviewing Slide 17

  18. ASHE Development 2004 Field tested 6 sided questionnaire: • Split sample design (98% ‘old’ , 2% ‘new’) • Quantitative analysis • Non-response • Respondent burden (actual & perceived) • Processing errors Slide 18

  19. Results of ASHE Field Test • Response (no statistical difference) • Errors (no statistical difference) • Respondent feedback (positive) • Respondent burden (reduced per question) • Some problems with question wording • Costs (increased paper, printing, postage, processing) • Time (increased processing time) Slide 19

  20. 6 sided to 4 sided compromises • Minimum standard – based on testing - keep question & response category wording + instructions to stay with questions • Challenge was to promote positive respondent perceptions and clear navigational path • No time for further testing – redesign by DCM experts & ‘quick perception’ telephone interviews Slide 20

  21. ‘Old’ Questionnaire Slide 21

  22. ASHE 6 Sided Questionnaire

  23. ASHE 4 Sided Questionnaire

  24. 2006 ASHE 4 Sided Questionnaire Slide 24

  25. 2005 Qualitative Follow-Up • Looked at differences between responses to the 2004 ‘old’ questionnaire and the 2005 4 sided questionnaire. • Focussed on 5 key areas e.g. • the affect on basic pay of a new ‘other pay’ question • 31 in-depth telephone interviews Slide 25

  26. ‘Old’ Basic Pay Question Slide 26

  27. ‘New’ Basic Pay and Other Pay Questions Slide 27

  28. ‘Other Pay’ Results Qualitative study found: • Better comprehension of new questions • Improved reporting of ‘other pay’ • Respondents preferred ‘new’ layout Quantitative analysis found: • ‘Old’ questionnaire - ‘other pay’ had been excluded by some • Correction by imputation added 0.8% growth to basic pay at the all employee level Slide 28

  29. What did we learn? Must have: • clear & agreed data requirements • agreement on the maximum number of pages • DCM experts involved in redesign & testing • method standards • design standards • improved methods for measuring data quality Slide 29

  30. Thank You Jacqui.Jones@ons.gov.uk

More Related