Web and Paper Questionnaires seen from the Business Respondent’s Perspective

1. Third International Conference on Establishment surveys. Montréal 2007 Web and Paper Questionnaires seen from the Business Respondent’s Perspective Gustav Haraldsen, Statistics Norway Jacqui Jones, UK Office of National Statistics

2. Outline • Aquality driven approach to response burden • The effect of survey design, external conditions and personal characteristics on perceived response burden • The impact of perceived response burden on data collection and data quality • (The perceived response burden of web versus paper questionnaires)

3. Origins and Effects of Perceived Response Burden in Establishment Surveys Response quality Survey Requirements Survey Design Response environment Context: Encoding/record formation Respondent selection/identification Priorities Sample Specification Frame Sampling procedure Size Stakeholder (s) PRB Questionnaire Length Structure Content Layout Survey organisation Procedure Contact mode(s) Persuasion strategies Management Confidentiality Release • Respondent(s) • Availability • Motivation • Competence Revision

4. The quality driven approach • Response burden is mainly a problem as long as it affects the response quality • It is the respondents’ response burdens that matters • Perceived response burden is a more relevant variable than completion time • The response burden may be affected by external conditions or personal characteristics of the respondents as well as by the survey design. • Perceived response burden questions should identify why questionnaires are burdensome

5. +1 +0.5 0 -0.5 -1 +1 +0.5 0 -0.5 -1 Core PRB Questions

6. Sources of perceived response burden • Survey Design Factors • Characteristics of 6 web questionnaires directed to different industries • (The time it took to fill in the questionnaire) • External conditions • Size: The number of employees • Information sources: Documentation problems that made the information collection time consuming • Division of labour: The number of persons who took part in the response process • Personal Characteristics • Availability: How easy or difficult it was to set aside time for the questionnaire • Motivation: How useful for the business and society the respondent felt that statistics produced from the data were • Competence: If he/she was a first time or experienced respondent

7. 2006 Web Response and PRB Response for Annual Business Surveys

8. Perceptions of the Information Collection and Completion of the 2006 Questionnaires Index Minutes 0,01 98 0,10 55

9. Sources of Time Consuming Preparations Other reasons Waiting for information Many questionnaires Need for help Different sources

10. Sources of Response Burdens E-technical Other E-usability Terms Layout No. of questions Response categories Mismatch Calculations

11. Information collection and Perceived response burden index: Industries and completion time

12. PRB-index: External conditions

13. PRB-index: Personal characteristics

14. The effect of survey design, external conditions and personal characteristics on PRB • Calculations and mismatch between questions and available information are general problems in business surveys • Sufficient time available for the questionnaire is essential for the perceived response burden. • The number of questions was probably the most important difference between questionnaires • Poor layout and usability problems were often mentioned by those who felt that the questionnaire was very burdensome • The quality of documentation systems is probably the most important difference between industries • The perception of uselessness goes together with perceived response burden

15. Web Response Patterns for Construction and Service Industry Questionnaires Service Construction

16. Questionnaires corrected

17. The impact of perceived response burden on data collection and data quality • Generally in business surveys.. • the data collection time is very long and • the number of questionnaires that are corrected is very high • When there is a pronounced difference in perceived response burden between questionnaires, there also seems to be a difference in… • How quickly the data are received • The number of questionnaires that need to be corrected • Problems with collecting the information asked for may also have an effect on these quality indicators

18. Perceived response burden of the paper and web versions of the Annual Manufacturing Industry surveys (2004 and 2005) PRB- index 0,01 -0,08 0,39 0,32

19. Sources of response burden in paper and web versions of the Manufacturing Industry Surveys

20. (The perceived response burden of web versus paper questionnaires) • When the Manufacturing Industry Survey changed from paper to web… • The perceived response burden apparently went down • While the number of problems reported by those who felt the questionnaire burdensome seemed to go up A quote for reflection: “It takes 100 men to do as much harm as a computer can make within 1/100 of a second”