Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau

1. Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison

2. Outline • Introduction & Background • Development of the Guidelines • Preliminary Guidelines • Application of Guidelines • Conclusion & Next Steps

3. Introduction & Background

4. Introduction & Background • U.S. Census Bureau: • “…leading source of quality data…nation’s people and economy” • Tries to minimize burden of data collection • Burden = time, level of effort • Reduce cognitive burden through visual design, structure of data request

5. Introduction & Background (cont’d) • Paper does not focus on specific question wording and order issues. • Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.

6. Development of the Guidelines

7. Development of the Guidelines: Census Bureau • Agency standards for various survey quality issues • No standard for design of questionnaires • Economic surveys moving toward consistent “look and feel”

8. Development of the Guidelines: ESMS • Establishment Survey Methods Staff • Group of survey methodologists • In-house consultants to economic programs • Noticed differences across questionnaires • Survey programs work with ESMS on questionnaire design and pretesting • Catalyst towards consistency

9. Development of the Guidelines: Methods • Questionnaire design principles from standard texts • Special attention to literature on visual design • Based on pretesting a variety of questionnaires with respondents

10. Development of the Guidelines: Considerations • List of design elements to consider • Used to assess tradeoffs within specific survey conditions • Not a cookbook

11. Preliminary Guidelines

12. Guideline A: Text styles • Be aware of text styles, and how they are used for emphasis within a survey instrument.

13. Guideline A: Example • 2004 Annual Survey of Local Government Finances

14. Guideline A: Example

15. Guideline A: Text styles • These are favored by respondents: • Print item numbers in reverse-print bubbles, e.g., , ,  • Print questions in bold black text. • Print instructions in plain text or italics. • Use an 8-point font or larger.

16. Guideline A: Text styles • Answer spaces: • Open, non-delineated vs. delineated • Respondents do not seem to have a strong preference • Be consistent

17. Guideline A: Text styles • Key codes / Punch codes: • De-emphasize processing codes for respondents • Print in a darker shade of background color • Example:

18. Guideline B: Reduce visual clutter. Use natural navigational paths. • Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.

19. Guideline B: Reduce visual clutter. Use natural navigational paths.

20. Guideline B: Reduce visual clutter. Use natural navigational paths. • Format in 1997 Economic Census confusing: • Lines were “speedbumps” to navigation • Not clear how respondent was to navigate two adjoined columns • Format changed for 2002 • Single column of questions • Generally, a single column of response options

21. Guideline B: Reduce visual clutter. Use natural navigational paths. • Two columns of questions • Occasionally done when questions: • Are shorter • Do not involve extensive instructions • Do not ask for numerical information • Survey of Business Owners (SBO) • Information collected is categorical or ordinal, not interval or ratio

22. Guideline B: Reduce visual clutter. Use natural navigational paths. • Spread out the questions across more pages • Use bulleted lists • More open space = more “user-friendly”

23. Guideline C: Instructions • Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

24. Guideline C: Instructions • Instructions convey specifications, intent of question • Respondents tend not to pay attention to instructions, or only look when they think they need them

25. Guideline C: Instructions • 2 Goals for instructions: • Eliminate, or reduce, amount of instructions located separate from question • Place instructions/information where it is most needed

26. Guideline C: Instructions • Convert instructions into questions so respondents attend to them • When content critical to correct interpretation of later questions • When it helps clarify/correct reported data • Example (2004 Medical Expenditure Panel Survey):

27. Guideline D: Avoid sentence fragments and key words • Phrase data requests as questions or imperative statements, not as sentence fragments or key words.

28. Guideline D: Avoid sentence fragments and key words • Types of data requests • Question (question word, question mark) • Imperative statement (report, enter, add) • Sentence fragment (key words, no verb)

29. Guideline D: Avoid sentence fragments and key words • Example, 2002 Economic Census: • Is this establishment physically located inside the legal boundaries of the city, town, village, etc.? • Versus: • Type of municipality where this establishment is physically located

30. Guideline D: Avoid sentence fragments and key words • Cognitive interview study: respondents prefer questions over sentence fragments • Survey methodology grad students: questions more effective, imply respondent has to do something

31. Guideline E: Simple vs. complicated questions • Ask additional, simple questions, rather than fewer, more complicated ones.

32. Guideline E: Simple vs. complicated questions • Length and complexity of question affects how long it takes for respondent to understand it • More complicated questions might also be more likely to be double-barreled • May be easier for respondents to answer series of shorter, simpler questions

33. Guideline E: Simple vs. complicated questions • Example: 2002 Industrial Research & Development Survey

34. Guideline E: Simple vs. complicated questions • 2006 R&D Survey

35. Guideline F: Use matrices judiciously • Use matrices judiciously, and consider the likely respondents’ background when deciding whether or not to use them.

36. Guideline G: Be consistent! • Items should be numbered in the same way • Bold and italicized print should always mean the same thing • Navigation path should remain constant • Data requests should be in the same form • Variation can be confusing for respondents

37. Application of Guidelines

38. Guideline A: Text Styles • Work in progress among estab surveys at Census Bureau • Economic Census automation required consistent design features • Plain text: questions • Italics: instructions, definitions, include/exclude lists • Bold: Separating items into categories, sums of added lines, emphasis within questions

39. Guideline A: Text Styles • Survey of Business • Owners: • Questions: plain • Instructions: bold and/or italics • Emphasis within questions: bold and/or italics

40. Guideline B: Reduce visual clutter. Use natural navigational paths. • Survey of Residential Alterations and Repairs (SORAR) • Two columns to one column • Removed or lightened lines • Use of bulleted lists • More open space

41. Guideline B: Reduce visual clutter. Use natural navigational paths. • Old • SORAR

42. Guideline B: Reduce visual clutter. Use natural navigational paths. • New SORAR

43. Guideline B: Reduce visual clutter. Use natural navigational paths. • Two columns: • Survey of • Business • Owners

44. Guideline B: Reduce visual clutter. Use natural navigational paths. • BEA quarterly foreign • direct investment (FDI) • Complex navigational • path • Crowded text

45. Guideline B: Reduce visual clutter. Use natural navigational paths. • BEA form • revised

46. Guideline C: Instructions • Economic census continues to use separate instruction sheets and booklets • Two questionnaires moved many instructions to questionnaire: • 2007 Commodity Flow Survey • BEA quarterly FDI form

47. Guideline C: Instructions

48. Guideline D: Avoid sentence fragments and key words • Mixed success in application across surveys

49. Guideline D: Avoid sentence fragments and key words • Commodity Flow Survey uses mix of questions and imperative statements

50. Guideline E: Simple vs. complicated questions • Example: • 2002 Survey • of Business • Owners