Diane K. Willimack U.S. Census Bureau

1. Introductory Overview Lecture Current Practices in Questionnaire Development, Evaluation, and Testing for Establishment Surveys Diane K. Willimack U.S. Census Bureau

2. The Response Process in Establishment Surveys and Implications for Pretesting Diane K. Willimack U.S. Census Bureau Survey

3. Outline • Establishments vs. households • The response process in establishment surveys • Questionnaire development, evaluation and testing methods for establishment surveys • Conclusions and challenges

4. Establishments vs. Households • What are establishments? • Economic units • Locations where business is conducted or industrial operations performed (FCSM, 1988) • How are establishment surveys different? • Population, data, and survey design • Response process

5. Establishment Surveys Are Different!

6. Establishment Surveys Are Different! Continued

7. The Response Process in Establishment Surveys Cognitive response model PLUS Organizational processes

8. Survey Tourangeau’s(1984)Cognitive Response Model • Comprehension • Retrieval • Judgment • Communication

9. Business Survey Response Process Model for Establishment SurveysSudman et al., ICES-2 Organizational Aspects • Comprehension • Retrieval • Judgment • Communication

10. Business Survey Response Process Model for Establishment Surveys Encoding in Memory / Record Formation Selection / Identification of Respondent(s) Assessment of Priorities (Motivation) • Comprehension • Retrieval • Judgment • Communication from Memory and / or Records Release of the Data

11. The Response Process in Establishment Surveys Encoding / record formation Respondent selection / identification Assessment of priorities Comprehension of the data request Retrieval of data Judging the adequacy of the response Reporting the response Release of the data

12. Records Memory Business Survey 1. Encoding / Record Formation

13. 1. Encoding / Record Formation, continued • Needs of company records • Management • Regulations • Standards – • Generally Accepted Accounting Practices (GAAP)

14. 1. Encoding / Record Formation,continued • Encoding includes – R’s knowledge of records encoded in memory

15. 1. Encoding / Record Formation,continued • If R’s knowledge of records is – • Flawed • Incomplete • Nonexistent • Response errors

16. Knowledge of Records Personal Knowledge Business Survey 1. Encoding / Record Formation,continued • In establishment surveys, encoding must include –

17. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

18. 2. Respondent Selection / Identification • Most knowledgeable R

19. 2. Respondent Selection / Identification,continued • Distributed knowledge • Multiple respondents • Organizational hierarchies • Authority • Respondent selection is under the control of the business / establishment.

20. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

21. 3. Assessment of Priorities • Financial reporting priorities • Requests from mgmt / investors • Annual Reports, SEC & IRS filings • Qtr & monthly financial reports • Regulatory requirements (govt & non-govt) • Other govt data requests (Census Bureau) • Everybody else (e.g., trade associations)

22. 3. Assessment of Priorities,continued • Financial reporting priorities • Job performance & evaluation criteria • Professional standards • Pride • R’s Motivation

23. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

24. 4. Comprehension • Interpreting the form • E.g., mailing label = reporting unit • Understanding the data request • Types and levels of data requested • Interpreting the meaning of the questions • Language, terminology, jargon

25. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

26. Personal Knowledge Records Knowledge of Records 5. Retrieval of Data • Data sources

27. 5. Retrieval of Data, continued • Retrieval incorporates – • Cognitive step – retrieving knowledge of data sources • Access to records • Physical act of retrieving data • Extracting information from files • Consulting multiple sources • Compiling information

28. 5. Retrieval of Data, continued • Different Rs differ in their – • Knowledge of records • Access to records • Ability to retrieve data

29. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

30. Records Memory Business Survey 6. Judging the Adequacy of the Response • Does R’s answer retrieved from records or memory match the data being requested?

31. Business Survey 6. Judging the Adequacy of the Response , continued • Does R’s answer retrieved from records or memory match the data being requested? • R doesn’t have the requested data in records. How does R estimate that figure?

32. Business Survey 6. Judging the Adequacy of the Response, continued • Does R’s answer retrieved from records or memory match the data being requested? • R doesn’t have the requested data in records. How does R estimate that figure? • R thinks, “My answer is good enough.”

33. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

34. 7. Reporting the Response • The act of writing the data onto the form or entering the data into an electronic instrument.

35. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

36. 8. Releasing the Data • Business relationships with the outside world • Review and verification of data • Reconciliation

37. 8. Releasing the Data, continued • Business relationships with the outside world • Review and verification of data • Reconciliation • Confidentiality • Security • Consistency and Sensitivity of Released Data

38. The Response Process in Establishment Surveys • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data • Judging the adequacy of the response • Reporting the response • Release of the data

39. The Response Process in Establishment Surveys:Personal Cognitive Steps+Steps at the Organizational Level • Encoding / record formation • Respondent selection / identification • Assessment of priorities (Motivation) • Comprehension of the data request • Retrieval of data from memory / records • Judging the adequacy of the response • Reporting the response • Release of the data

40. Some General Characteristics of Establishment Surveys • Complex response process • Request technical, records-based data • Skewed target populations • Estimate change vs. level • Emphasis on timeliness of reporting

41. Emerging Themes That Affect Establishment Survey Pretesting Nature of requested data Survey response is labor-intensive   Uses of economic data Respondent burden  

42. Questionnaire Development, Evaluation and Testing (QDET) in Establishment Setting • How QDET for establishment surveys has addressed – 1) Nature of the requested data 2) Survey response is labor-intensive 3) Respondent burden 4) Uses of economic data

43. 1) Nature of the Requested Data

44. 1) Nature of the Requested Data:Issues for QDET • Technical concepts with precise definitions • Data found in business records

45. 1)Nature of the Requested Data Technical Concepts with Precise Definitions • Personnel involved in questionnaire development • Subject area experts • Stakeholders and data users • Questionnaire design experts

46. 1)Nature of the Requested Data Technical Concepts with Precise Definitions (cont.) • Consultation methods • Expert / User / Advisory Groups • Iterative rounds of prioritizing data needs • Large-scale content reviews • Exploratory focus groups • Observers during pretesting

47. 1)Nature of the Requested Data Technical Concepts with Precise Definitions (cont.) • Standardization and harmonization of questions and concepts • Instructions

48. 1)Nature of the Requested Data Data in Records • In order to design effective questionnaires, information is needed about – • Data availability • Degree of comparability • Estimation strategies • Ease of retrieval • Respondent identity

49. 1)Nature of the Requested Data Data in Records(cont.) • Methods • Exploratory / feasibility studies and site visits • Small, purposive samples • Meetings with company staff • Topic protocol • Record-Keeping Studies

50. 2) Labor-IntensiveSurvey Response Process