1 / 21

Slide 1

Quality of Pretesting: Instruments for Evaluation and Standardization Session 23: Survey measurement issues Q2010 in Helsinki May 3-6, 2010. Slide 1. Contents. Pretesting at the FSO Quality standards in qualitative pretesting Future prospects. Contents. Pretesting at the FSO

devlin
Download Presentation

Slide 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quality of Pretesting: Instruments for Evaluation and Standardization Session 23:Survey measurement issues Q2010 in Helsinki May 3-6, 2010 Slide1

  2. Contents • Pretesting at the FSO • Quality standards in qualitative pretesting • Future prospects

  3. Contents • Pretesting at the FSO • Quality standards in qualitative pretesting • Future prospects

  4. Institutional background • Code of Practice (2005), principle 8:“Questionnaires are systematically tested prior to the data collection.” • Eurostat QDET (2006): systematic testing in the following cases • a new survey • new or modified questions • additional or modified data collection instrument • poor data quality Pretesting: • Increase in data quality • Decrease in respondents’ burden

  5. Methods of pretesting • Quantitative testing methods: • Multitude of probands (N > 100) • Under field conditions • Frequency of problems with the questionnaire • Qualitative testing methods: • Limited number of probands (N ≤ 20) • Under laboratory conditions • Reasons for problems with the questionnaire • First ideas for improvement  Three step approach

  6. Step I: Observation • Sources of information: • Gestures, facial and short verbal expressions (“reality without words”) • Remarks in the questionnaire • Gain of knowledge: • Independent and unaffected behavior without any advance information

  7. Comprehension Information Retrieval Judgment Response Step II: Cognitive interview • Sources of information: • Insights in the response process by the use of cognitive methods • Narrative description of personal situation • Gain of knowledge: • Reasons for incorrect or missing answers • Individual reality  questionnaire • Suggestions for improvement (Tourangeau/Rips/Rasinski 2000)

  8. Step III: Evaluation of the questionnaire • Sources of information: • Entries in the questionnaire • Remarks, question marks, etc. • Gain of knowledge: • Actual handling of the questionnaire beyond what respondents thought they had understood

  9. Contents • Pretesting at the FSO • Quality standards in qualitative pretesting • Future prospects

  10. Need for quality standards • Qualitative methods are often criticized as being unreliable, unrepresentative and insignificant • Statistical offices traditionally work quantitatively  new development to elaborate standards for qualitative data and to improve their explanatory power

  11. Criteria for high quality of qualitative data • Checking for generalization without verification • Checking for representative probands • Checking for researcher effects • Triangulation • Balancing the evidence (Miles/Huberman 1994)

  12. Checking: generalization without verification • Avoid to regard conclusions for one or two very striking probands as typical (“You see what you want to see.”) • Safeguards: • Consider positive and negative evidence • Quantify qualitative data by the use of QDA software and matrices • Double-check codings and conclusions in team

  13. Checking for representative probands • Approximately 20 probands who represent the ordinary respondent in official statistics; group shall be as heterogeneous as possible • Safeguards: • Select probands adequate for the target population • Invite probands with different social background by different ways of recruitment • Establish a data base with information on probands

  14. Checking for effects on probands • Intimidatedby the testsituation • Social desirability or acquiescence • Concerns about providing information to the “government” • Safeguards: • Create a comfortable atmosphere • Warming-up (course of the test, expectations towards the probands) • Underline anonymization and confidentiality

  15. Checking for effects on interviewer • Leading questions • Losing distance(”going native“) • Safeguards: • React in an adequate way, remain neutral • Avoid additional remarks on personal opinion or survey question • Ask for mutual feedback in team

  16. Triangulation • Confirming results by replicating them • Taking different perspectives on the questionnaire • Gain an overall picture • Safeguards: • Data triangulation (probands, places, points in time) • Researcher triangulation (teamwork) • Methods triangulation (three step approach)

  17. Methods triangulation Questionnaire Overall picture Observation Cognitive interview

  18. Balancing the evidence • “Stronger data can be given more weight in the conclusion.” (Miles/Huberman 1994) • Safeguards: • Make a note of cases with poor data quality • Remember theses cases during data analysis • Exclude these cases from the final report, if necessary

  19. Contents • Pretesting in official statistics • Selected results • Future prospects

  20. Future prospects • Quality standards for qualitative pretesting (e. g. checklists) • Online questionnaires • Business statistics • Elaborated guidelines for cognitive interviewing • Exchange of experience between statistical offices

  21. Thank you for your attention. sabine.sattelberger@destatis.de simone.tries@destatis.de

More Related