IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement

1. IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement Pete Brodie ONS, UK

2. Overview • Background • Drivers • Questionnaire requirements • Delivery dilemma • Solution and outcome • New output requirement • “Agile” methodology • Outcome • Results • Next steps • Conclusion

3. Background • ONS delivers a very broad spectrum of statistics • Financial pressures on all NSIs • Difficult to react quickly to new requirements • Scarcity of specialist skills

4. Drivers • Changes due to ESA 10 and BPM6 - updated regulations for producing National Accounts • Changes in the labour market – the way people are employed

5. Questionnaire requirements • Early in 2014 legislative changes required substantial changes to most ONS business statistics • First tranch of 28 Finance Surveys • Total redesign of questionnaires • Additional questions • Covering new sectors of the economy

6. Delivery dilemma • Standard approach for DCM uses multiple waves of cognitive interviews with responders • Transcription of interviews • Analysis using thematic framework • Detailed report of issues and recommendations • Time and available resource – impossible!

7. Solution and outcome I • Brand new approach needed trialled for two surveys • Condensed usual waves of testing into one week “stints” • Three stages to each stint

8. Solution and outcome II • Traffic light document

9. Solution and outcome III • Extensive changes implemented for two surveys – agreed between all stakeholders • Real improvements made • All questions rated “green” at end • Client feedback very positive • Re-use methodology with some tweaks

10. New output requirements • Autumn 2013 demand for measurement of the number of employees on “Zero Hours” Contracts • Definition unclear • No information from businesses • Promise to produce estimates early in 2014 • Conventional approach of qualitative and quantitative research likely to take around three years

11. “Agile” methodology I • Staged approach – pilot followed by live survey • Combined qualitative and quantitative work • Novel (for ONS) approach of telephone survey of businesses • Used flexible “Statistical Computing” for all quantitative production

12. “Agile” methodology II • Pilot survey of around 1000 businesses (30 strata) • Supplemented frame information with contact numbers • Collaborative working between different parts of the office • In field for four weeks achieved 60% response • Qualitative interviews with selected responders • Bespoke code for editing, outlier detection and estimation

13. “Agile” methodology III • Quantitative estimates from pilot used to optimise survey design for main survey – aim for 10% cv at top level • Questionnaires updated using qualitative research • Main survey of 5000 businesses included all pilot sample and 4000 new members • In field for six weeks achieved over 60% response

14. Outcome • Published results on 30th April – first business survey estimate of the number of live contracts of 1.4 million (c.v. 8.5%) • Greater understanding of term -“contracts with no guaranteed minimum hours” • Comparison with estimates from Labour Force Survey • Published some more detailed estimates • Some data available for longitudinal analysis

15. Results I Proportion of business using ZHC by size

16. Results II Proportion by broad industry

17. Next Steps • The LFS questionnaire was updated in the light of findings and to answer detailed demographic questions • Follow up business survey of 5000 now in the field • Considering longitudinal design • Further embedded qualitative work • Considering longer term strategy

18. Conclusions • These two examples illustrate what is possible when using an agile approach. • Too often constrained by artificial boundaries. • Opportunities for development of staff and methods through collaborative working. • Necessity is the mother of invention!

19. Thank you! Any questions ?