1 / 43

Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs WP4

This workshop discusses the recommendations for developing mixed-mode questionnaires in social surveys, with a focus on Eurostat and NSIs. Key topics include questionnaire design approaches, feasibility, survey practices, and deliverables.

jangeline
Download Presentation

Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs WP4

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs WP4 Dag F. Gravem Statistics Norway

  2. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Agenda Overview of WP4 Presentation of deliverables Recommendations for discussion Discussant Anette Björnram, Statistics Sweden Open discussion MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  3. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Overview of WP4 • Main contributors • Statistics Norway (WP leader) • CBS • Istat • Supporting contributor • Destatis MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  4. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Overview of WP4 • Recommendations on approaches for developing web questionnaires for mixed-mode surveys • Questionnaire level • Question level • Questionnaire elements • “Unimode vs. mode-specific questionnaires” MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  5. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Overview of WP4 • Feasibility of mixed-mode questionnaires and usefulness of MIMOD recommendations dependent on several factors • Survey practices and infrastructure • Survey communication options • Survey content and requirements (from Eurostat) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  6. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs WP4 Deliverables • A paper on mixed mode combinations and experiences at participating NSIs • A paper on survey communication in mixed-mode ESS surveys • A paper with recommendations for key questionnaire elements, questions and question types in mixed mode settings • Methodological report MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  7. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs WP4 Deliverable 1 • Mixed mode experiences at European NSIs • Data from MIMOD survey • Mixed mode combinations in use • Questionnaire differences • Opinions on surveys not fit for CAWI • Good and bad practices • National case studies • Stats Norway: Mixed mode in a historical perspective • Istat: Recent experiences from census and aspects of life survey • CBS: The “Omnimode” questionnaire design system MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  8. Deliverable 1: Mixed mode combinations in use

  9. Deliverable 1: Concurrent, sequential & combined mixed mode data collection

  10. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs • Deliverable 1: Questionnaire differences • Error and consistency checks: many, but small • Don’t know and Item nonresponse: many, and some large

  11. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs • Deliverable 1: Fitness of key ESS surveys for CAWI • LFS: Lack of interviewer support • EU-SILC: Length of survey MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  12. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs National case studies: CBS • CBS guidelines for omnimode design • Keep the stimulus – question and answer categories – the same across modes • Redesign using unimode principles if possible. Don’t use one mode as the primary one. • Use a screen-by-screen design to keep the respondent focussed on one question at a time … • Unless they are filter or follow-up questions • Repeat question texts on next page/screen to give the complete stimulus again • Always include all answer options in questions to be read out loud, including No opinion, Refused or Don’t know. • Avoid grids if possible, and ask series of questions in all modes instead • Minimize instructions and explanations, and present them similarly across modes, preferably in the form of questions

  13. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Deliverable 3: recommendations for key questionnaire elements, questions and question types in mixed mode settings • Theoreticalframeworks for mixed mode analysis • Assessmentofquestionnaires • Eurostat modelquestionnaires and documentation • National implementations • Results from prestesting performed for MIMOD • ICT, EHIS, EU-SILC, AES, LFS • Appendix: Test report from Statistics Norway MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  14. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Deliverable 3: Theoreticalframeworks • Unimode, mode-specific, generalized mode, mode enhanced design • De Leeuw & Hox: Different design conventions – question format effects • Often mode specific designs by survey practicioners • Experimental designs often try to minimize question differences MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  15. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Deliverable 3: Theoreticalframeworks • MIMOD predecessor DCSS (Körner et al. 2013) • Key factors for differentialmeasurement • Type ofsocialinteraction – interviewerinvolvement, respondent control • Type ofcommunication – verbal, non-verbal, para-verbal, computer mediated • Questionnaire design options – visual vs. aural stimulus • Computer assistance • High risk question types • Sensitive, difficult, long, manyresponseoptions, complex skip instructions MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  16. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Deliverable 3: Theoreticalframeworks • Campanelli et al. (2013): QuestionCharacteristics Relevant to MeasurementError • Mode recommendations for 29 different characteristicsofindividual questions grouped by • Questioncontent – topic, sensitivity, inherent difficulty • Question format – open, closed, ratio, ordinal, nominal • One questionmay have several and conflictingcharacteristics MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  17. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs

  18. Tableof moderecommendations

  19. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Deliverable 3: Campanellicontinued • Question types notrecommended for all modes • 1) Sensitive questions • 4) Subjective, non-sensitive scalar questions • 6) Unconstrained textual/verbal open questions • 10) Open questions with interviewer coding • 14) Mark all that apply response format • 16) Ranking questions • 18) Visual analogue questions • 20) Questions with a high number of response categories • 24) Questions with edit checks • 29) Questions with show cards MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  20. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Deliverable 3: Mixed mode pretesting by Statistics Norway • Test-retest design • Test subjectswere first interviewed in CATI mode • Thenresponded to the same questions in CAWI mode about a week later • Retrospectiveprobing • Eyetracking • Test ofunimode design: Entire ICT questionnaire • Test ofmode specificdesign for problematic questions according to Campanellicriteria: • Selected questions from ICT, EHIS, EU-SILC, AES and LFS surveys MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  21. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: ICT questionnaire • ICT survey regulationcontainsnorecommendations or limitationsregardingwhich mode to use • But: Model questionnairepresupposes a visual format • But CATI most used (16 NSIs) • Followed by CAWI (15 NSIs) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  22. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs ICT modelquestionnaireexample

  23. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires:ICT questionnaire • Application ofCampanellitypologyonmodelquestionnaireindicatespossibleissues A. Mark all that apply format (14) B. Inherent difficulty due to concepts (6) C. Use of instructions and clarification (23) D. Use of Don’t know (25) E. Questionlength (Dillman) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  24. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs ICT: National implementations • Netherlands • CATI/CAPI/CAWI design • Omnimodeapproach: everything visible on a screen will be readoutloud • Norway • CATI only • Embedded in 35-minute omnibus • «Check all thatapply» grids converted to yes/nosequences MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  25. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs ICT questionnaire Pretesting • Stats Norway: Both unimode (omnimode) and mode specificapproachtested • Unimode: entirequestionnaire • Mode specific: selectionswith «mark all thatapply» and inherentlydifficult questions • Mobile unimode basedonexisting CATI questionnaire • PC mode specificversionbasedonmodelquestionnaire MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  26. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs ICT pretesting: mainfindings • Unimode: General problems more importantthan mode-related problems • Overall verbosity – redundant phrases • Insufficientoperationalization – tootheoreticallanguage • Transformation to yes/noincreasesquestionnairelength and makes it difficult for respondents to have an overview • Mode specific PC: • Indicatethat «check all thatapply» on PC can be combinedwithyes/nosequence in CATI for the questions thatweretested • Should be testedquantitatively MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  27. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: EHIS • EHIS: face-to-face modes described as «preferred modes» in methodological manual • Show cardssuggestions • Separate documenton CATI implementationof questions onalcoholconsumption • CAPI used by 15 NSIs, CAWI by 11 • Note: literaturedoesnotrecommend face-to-face for personal interviews MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  28. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: EHIS • Application ofCampanellitypologyonmodelquestionnaireindicatespossibleissues A. Sensitive questions (1) B. Long “yes/no for each” question batteries (15) C. Show cards (29) D. Inherent difficulty due to difficult terms and calculations (5) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  29. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs EHIS question pretesting • Questiononalcoholcomsumption • Mode-specific test-restest • CATI: sequenceof questions • CAWI: Single questions withnineoptions • Indicationofequivalence • But 9 optionstoomuch for mobile ..? MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  30. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: EU-SILC • Methodological guidelines statethatfive modes of data collectionarepossible, PAPI, CAPI, PASI and CAWI • «Priority to be given to personal interviews» • CATI not covered in legal basis, butallowedon «gentleman’saggreement» in case of person ratherthan household samples • Recommendationspresupposeinterviewer-administered modes: No discussiononhow to implement for self-administered • CATI and CAPI most widely used MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  31. EU-SILC pretesting • Mode-specific test-retest of questions onmortgages • Interviewer for motivating and assisting in CATI • Visual presentation and automaticcalculations in CAWI • Inspired by business survey questionnaire design MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  32. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs EU-SILC questionnaire prestesting • Visual presentation: increasedperceivedresponseburden • Satisficing • Calculationswithsubtractionsnot intuitive • Measurementerror • Layout not suited for mobile screens • Device effects in addition to possible mode effects? MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  33. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: Adult Education Survey (AES) • Reference questionnaire and survey guidelines areavailable • Variable INTMETHOD has ninevalues • Pros and consof different data collectionmethodsarediscussed • CAPI and otherinterviewer-administered modes recommended in guidelines • Butreferencequestionnaireoftenpresupposes a visual mode • CAPI most used mode (17), followed by CAWI (11) and CATI (10) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  34. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires:AES questionnaire • Application ofCampanellitypologyonmodelquestionnaireindicatespossibleissues A. Mark all that apply format (14) B. Inherent difficulty due to long and complex questions (5) C. Inherent difficulty due to long recall period (5) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  35. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs AES pretesting • Mode-specific CAWI test of questions onhoursof training • Visual presentation and automaticcalculations MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  36. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs AES questionnaire pretesting • Non-mode specificissuespersist • Whatshould be consideredonelearningactivity? • Should breaks be included or excluded? • Difficultiescalculatingaveragelengthofdays • Calculatordoes not solvethese problems • Canevenenhancethemwith an extrataskofevaluation MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  37. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: LFS • No modelquestionnaire, but minimum reqirements in terms of variables found in explanatory notes • No mentionof CAWI • Show cardsmentioned as the «essentialdifference» between CAPI and CATI … • CAPI most common data collectionmethod in LFS W1 (22) • CATI most common in W2+ (15) • CAWI consideredunsuitable for LFS W1 by 20 NSIs • Recruitment and identification • Need for interviewerclarification MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  38. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessmentofquestionnaires: LFS • Application ofCampanellitypologyonmodelquestionnaireindicatespossibleissues A. Inherentlydifficult questions – includingrecall (5) B. Open questions withinterviewercoding (10) C. Useofinstructions (23) D. Show cards (29) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  39. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs LFS pretesting • Mode-specific test-retest ofquestiononactualworkinghours • Interviewer for motivating and assisting in CATI • Visual presentation and automaticcalculations in CAWI MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  40. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs LFS prestesting • Similarsetuptestedpreviously • Statistics Finland in the DCSS project – positive pilot results • Equivalentresultswith CATI • ONS – negative results from cognitive and usability tests • Annoys respondents • Satisficing • Stats Norway’sresultsverysimilar to UK experience • E.g. roundingoff to full hourseveryday (seefigure) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  41. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Recommendations and conclusions for discussion • Difficult to offer recommendationsthatcontradictexisting Eurostat requirements and recommendations • Rebuild all model and nationalquestionnaires and documentationwithmixed mode in mind – modes will be mixed! • Start from scratchinsteadofcontinuingon an existingpatchwork • Shorten, modularize, simplify • Take unimode as a startingpoint • Butcontinueexploring mode specificsolutionsusingcognitive and usability testing, pluspiloting MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  42. Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Recommendations and conclusions for discussion • The Campanellitypology: useful, butneedsupdating • Little in terms of CAWI specificrecommendations • Othertools for identifying and analyzingproblematic questions? • Better facilitationof rapid exchangeofexperiences and recommendations • Includingmixed mode pretesting protocols • Risk ofbeingoutdatedbeforepublication • Documentationof surveys and questionnairefragmented • A bettertoolthanCros-portal and Circabc? • Wiki-based web page? MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

More Related