1 / 18

A generic tool to assess impact of changing edit rules in a business survey – SNOWDON-X

A generic tool to assess impact of changing edit rules in a business survey – SNOWDON-X. Pedro Luis do Nascimento Silva Robert Bucknall Ping Zong Alaa Al-Hamad. Business survey editing in the ONS. Uses complex sets of edit rules to: Check returned questionnaires (records)

russell-ryan
Download Presentation

A generic tool to assess impact of changing edit rules in a business survey – SNOWDON-X

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A generic tool to assess impact of changing edit rules in a business survey – SNOWDON-X Pedro Luis do Nascimento Silva Robert Bucknall Ping Zong Alaa Al-Hamad

  2. Business survey editing in the ONS • Uses complex sets of edit rules to: • Check returned questionnaires (records) • Locate suspicious or unacceptable responses • Support data cleaning operations • Edit sets are complex because they may: • Involve a large number of survey forms and variables • Contain a large number of edits • Define complex acceptance / rejection regions • Depend on a large number of tolerance parameters

  3. Editing costs are high • The estimated cost of editing is over 40% of the survey process budget • Edits cause large numbers of record failures • Edit failures are mainly dealt with by means of manual follow-up, re-contacting respondents

  4. Aim of paper • Describe a generic tool developed to assess the potential impact of changing the edits in any specified business survey • Present example of application of the tool (SNOWDON-X) to large scale annual business survey

  5. Edit revision strategies for efficiency saving • Filtering or sub-setting • Comprises introducing a record filter which selects the records to be submitted to the full set of edits • Gate widening • Consists of revising the tolerance parameters (gates) in individual edit rules, such that flagging of suspicious records for revision is less frequent than with previously used values • Edit deletion • Consists of simply discarding some of the edits previously used to flag suspicious records

  6. SNOWDON tool • A SAS program developed first by Al-Hamad, Martín and Brown 2006 • Developed to enable informed decision making when revising business survey edits • Aims to “… help survey managers evaluate what savings can be achieved, at what cost to output quality, across many alternative permutations of editing rule parameters.” • Limited to single variable survey, where only ‘gate-widening’ was considered

  7. SNOWDON-X tool • Extended funcionality when compared to SNOWDON • Uses SAS IML language for increased performance • Can handle all three edit revision strategies • Can handle multivariate surveys • Provides a wealth of summary indicators relating to: • Expected savings achievable by edit revision • Expected bias to survey results, both overall and per variable • Information on performance of individual edit rules / variables • Simple to run, once data have been properly organised

  8. Basic scenario • Previous survey data available in two versions • Unedited (raw) – at point of capture or prior to any editing • Edited (clean) – at point of publication or after all editing • Edit rules used to clean previous survey data are known • Key idea of SNOWDON-X tool • Increase tolerance of some edits (or delete or introduce filter if necessary) • Calculate indicators of impact of changes to edits • Repeat 1. and 2. until expected savings achieve specified level or quality measures reveal unacceptable bias

  9. Key assumptions behind approach • Future survey edition will behave similarly to previous survey • Edited data from previous survey edition are ‘clean’ or error free • Changes to ‘raw’ data in previous survey edition were due to error correction, i.e., any values changed between capture and final were ‘wrong’ • Once a record is flagged for clerical revision, all errors it contains will be located and corrected

  10. What is required to run SNOWDON-X?

  11. What is the output of SNOWDON-X?

  12. Core indicators

  13. How to target edits for revision • Select most commonly used form type • Select edit failing largest proportion of records within each form type • Relax edit parameters to reduce proportion of failed records while keeping bias low • Repeat 2. and 3. for each form type until further savings are minimal or bias increases above specified threshold • Repeat for all relevant form types

  14. ABI/2 (Retail questionnaire) – Number of failing records on original and revised edits

  15. Results - applying SNOWDON-X to ABI/2 (Retail questionnaire)

  16. Results from applying SNOWDON-X to ABI/2

  17. Results summary • Overall expected saving for ABI/2 ≈ 6% of previously edited records • Largest expected bias occurs in Catering sector (0.65%) where a saving of 58 (9.6%) records was made • Highest expected saving was made in Motor Trades sector (11.1%), with an expected bias of 0.31%

  18. Conclusions • Generic tool developed to assist edit revision • Successfully applied to ABI2 • Currently being applied to two monthly surveys • SNOWDON-X tool enables focus on edit revision, not programming for calculating quality and savings indicators • Further development required for: • Impact on standard error estimates • Improved usability

More Related