Questionnaire design

1. Questionnaire design

2. Presented by • Paweł Lańduch • Central Statistical Office, Statistical Office in Poznań, Poland.

3. Questionnaire design Modules • Maintheme, • Electronic questionnaire design, • Editingduring data collection, • Testingthequestionnaire .

4. Questionnaire design • Business surveys – special population • the response process is complex and burdensome, since preparing the required data entails a mixture of organizational and individual tasks. Data are mostly quantitative and their acquisition requires access to business records; • the predominant role of self-administered questionnaires and the role of the respondent. In the case of establishments, the respondent is usually a representative, who acts as a data provider in an organizational environment,

5. Questionnaire design • Business surveys – special population • the mandatory character of reporting as opposed to mostly voluntarily aspect of household surveys, • concepts and definitions are technical, complex, and often based on legal or regulatory considerations. Practices, terminology and standards used by businesses in their daily operations, the accounting standards, for example, are the context that needs to be taken into account. This calls for detailed instructions that accompany questions.

6. Questionnaire design • Business surveys – special population • the distribution is skewed in favor of larger establishments, • timeliness is often given priority over quality. • the longitudinal character of surveys and overall reluctance to changes in measurement instruments used.

7. Questionnaire design • Responseprocess Definition - the result of the interaction between a respondent and aquestionnaire (Edwards & Cantor, 1991, Measurement Error in Surveys) Housholdsurveysversus business surveys individualorganisation usingmemoryusingrecords cognitioncognition + organisation retrievalretrieval + recordslookup

8. Questionnaire design • Concepts and variables (example) Short term statistics • Term – Industrial production sale value in current prices (Sa) • Variables • Revenues from products sale (Sw) • Value of goods production not including in sales (Sz) • Excise imposed on products (Da) • Subsidy for products (recevied for products)(Dp) Sa=Sw+Sz-Da+Dp

9. Questionnaire design • Choosing the mode of data collection • Interviewer administered • Self-administered • Mixing modes • Impact of computer technology – web forms • Actual burden versus perceived burden • Numerical character of data • One of the most burdensome phase – data retrieval

10. Questionnaire design • Building the prototype questionnaire • Content and non-content elements • Introductory letters, • Cognitive aspects • Layout, • navigation • instructions • technological challanges • Heavy reliance on instructions • Guidelines (examples: Economic Directorate Guidelines on Questionnaire Design. Washington, DC: U.S. Census Bureau (2008).

11. Questionnaire design • Guidelinescont. examples: Forms Design Standards Manual, Australian Bureau of Statistics (2010), • Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System,Eurostat (2006) • Creatingstandards as a framework for developing, testing and evaluation, also for technicalaspects • Example (Web environment – hollisticview, dynamics, one point for collection, one-size-fits-all ?) • Measurementerrors-QualityCost-ResponseBurden-Usersatisfaction

12. Questionnaire design – Electronic questionnaire • Impact on responseprocess • Impact on responseburden • Impact on quality • New features and opportunities but alsochallages • Electronic versus paper Layout – paging, scrolling, shouldthe electronic versions be smiliarinappearance to those on paper ? Editing – movingcloser to the respondent, costreduction, reducingburden, seekingbetterquality

13. Questionnaire design – Electronic questionnaire cont. Automatic routing – eliminate routing errors, previous answers determine next questions presented, Calculations – expecting by respondents, are elements of data consistency and validity Progress indicator – possibility of checking the completion status, there are also arguments against placing it Navigation – seeking easy way for moving forward and backward, possibility of interutption and resuming Instructions – close at hands, examples: hyperlinks and pop-ups Security – important for respondents espacially in busines surveys

14. Questionnaire design – Electronic questionnaire cont. Testing for usability - Gestaltpsychology for visual design - Cognitivepsychology - designer’sperspectiveversususer’sperspective - user-centre design - designing for errors - visibility, mapping and feedback - standardizing - heuristicapproach (Jacob Nielsen)

15. Questionnaire design – Electronic questionnaire cont. Example: paper questionaire and its electronic counterpart

16. Questionnaire design – Electronic questionnaire cont. Microsoft Office as an example of standardizing a toolbarin a suite of programs

17. Questionnaire design – Electronic questionnaire cont. The effect of automatic routing (the gap in numbering)

18. Questionnaire design – Editing during data collection • Possibility to movesome of editingactivities (so far as a post-collectionprocess) on collectionstage. • Reduction of costs and data qualityimprovement • Potentials for loweringburden • Build-ineditingrulescallededitchecksoredits – a respondent isnotifiedabouterrorswhileenteringresponses • Types of implementededitrules – hardedits and soft edits Thereis an expectationcollected data are of high quality and contain no errors – THAT MAY NOT BE TRUE

19. Questionnaire design – Editing during data collection Presenting messages to respondents (examples from CSO of Poland) Using icons in messages Marking hard edits failures (errors) Soft edits (warnings)

20. Questionnaire design – Editing during data collection Text of message as a pop-up box or in a list (examples) Red text: Error: The information you provided does not match our criteria.

21. Questionnaire design – Editing during data collection Varioustypes of edits for varioustypes of items: • Text – length, format check • Numbers – range, algebraicalrelations, balancecheck • a reponsechoosingfrom a predefined list • Requireditemcheck • A ruleallowingonlysome sort of characterse.g. numeric • Checkingtherelationshipbetweentwoormoreitems

22. Questionnaire design – Editing during data collection Considerations when choosing a strategy for edit rules in questionnaires • How many edit rules will be effective and not resulting in item non-response ? • Conservative aproach versus extensive usage of edit rules • Which way of preseting messages is efficent to be recognized by respondents ? • Even if the messages are perceived they should be understood

23. Questionnaire design – Editing during data collection Considerations when choosing a strategy for edit rules in questionnaires (cont.) • How data with unresolved edits should be treated ? • How to test and evaluate edits in the questionnaire ? • Should be permitted to submit data with unresolved edits ? Some hints from usability testing: • user should be at the center of design, the task should be under the user’s control • Words „error” and „mistake should be avoided

24. Questionnaire design – Editingduring data collection Somehintsfromusabilitytesting (cont.): • Edit rulesshouldbe triggeredimmidiately as theycan, but with a possibilty of a repeteadexecution, • Themessageshould be providedwithattributes for easylocalization of theproblematicitem • Clear signification to the respondent theobligation to resolvealleditsor to provide a commentbeforesubmission • Deliberationbeforeproviding data fromearlierperiods to avoidacquiescencebias

25. Questionnaire design – Testingthequestionnaire • EuropeanStatisticsCode of Practice, Principle 8, Indicator 8.2: In thecase of statisticalsurveys, questionnairesaresystematicallytestedprior to the data collection. Thecontinuousprocess • A draft questionnairereviewpretestingrevision • Data collectionevaluationrevisionpretesting • pretesting / field testing

26. Questionnaire design – Testingthequestionnaire • CASM Cognitive aspects of survey methodology • „thesubject” thoughtprocess of response ThinkaloudVerbal probing Socialsurveys Business surveys at business location presurveyvisits expansion of cognitiveinterviewing

27. Questionnaire design – Testing the questionnaire Recordkeepingstudies • One of the most burdensome step intheresponseprocess – retrieval of thedemanded data • Data may not existin business records • Data may be keptindifferentplaces • Cognitivebackground • Differentpersonswithvariousdegree of access to data

28. Questionnaire design – Testing the questionnaire Problemswith data retrieval estimationroughestimationblundernon-response Usabilitytesting Visual aspect • Fonts • Graphics • Visual clutter Userinterface • Navigation • Skippingroutes

29. Questionnaire design – Testing the questionnaire - Usabilitytesting (cont.) Software theprocess (planning, analysis, implementation, assesement, closure) functional tests (black box) structural tests (white box) - Testinginthe field (after data havebeencollected) Pilot tests – dressrehearsal Debriefingrespondents – formal and informal - thesource of informationaboutthequestionnaire Debriefingsurveystuff – formal and informal – as intermediariesbetweenrespondents and surveyagencyhavevaluableknowledge for thequestionnaireevaulation

30. Questionnaire design – Testingthequestionnaire • Testinginthe field (after data havebeencollected) Post-collection data evaluation – a tool for assessing data quality, • Measuringnon-response • Detectingoutliers • Data editingfailures • Comparingwith data fromothersources

31. Questionnaire design – Testingthequestionnaire

32. Questionnaire design – Testing the questionnaire • Settingstandards for testing • Minimalstandards • Extendedstandards • Best practices • Responseprocess as a tool for thequestionnaireevaluation • Based on steps of theresponseprocessin business surveys • Based on a topic list of questions • Qualitativemethod for improvements

33. Thankyouvery much for yourattention