La Toya Barnett Thomas Rebecca L. Morrison Grace E. O’Neill U. S. Census Bureau

1. Cognitive Aspects Associated with Sample Selection Conducted by Respondents in Establishment Surveys La Toya Barnett Thomas Rebecca L. Morrison Grace E. O’Neill U. S. Census Bureau

2. Outline • Background • Methodology • Cognitive requirements of selecting a sample • Results • Conclusion • Future Research

3. Background: CFS Information CFS = Commodity Flow Survey. Mandatory and collected every 5 years. Last survey covered 2002, next survey to cover 2007. Collects information on shipment of goods across the country for a 1 week reference period. Unique Feature: respondents required to select their own sample to complete form.

4. Background: The Issue Asking non-statisticians to perform a major statistical task – systematic sampling. Not a normal survey task in general. Respondents drawing their own sample was an issue in 2002 survey. Sponsors suggested cognitive testing in preparation for 2007. Sample selection covered in 3 sections (Items D, E, and F) of the survey.

5. Background: The Issue (cont’d) Item D –total number of outbound shipments during reference period. Item E – guides R’s through process of selecting their sample, based on their response in Item D. Item F –information about shipments from their sample (dates, type of shipment, destination, how transported).

6. Methodology Three rounds of cognitive interviews (October 2005 – August 2006) Nationwide 64 interviews total Companies were a mix of industries, sizes, products, and shipping arrangements.

7. Methodology (cont’d) • Tourangeau’s (1984) Four Step Cognitive Response Model • Comprehension • Retrieval • Judgment • Communication

8. Cognitive Requirements of Selecting a Sample: Comprehension Understanding navigation and meaning of words and/or diagrams. Getting lost in navigation causes difficulty in completing task. Basically, if Rs lack comprehension, the task of selecting their own sample will be difficult.

9. Cognitive Requirements of Selecting a Sample: Retrieval Accessing several types of records: • Fleet of vehicles and/or trucks • Parcel delivery services • Customer pick-ups Records could be on paper, in electronic form, or both. Records in multiple places – need to know that they exist and how to access them.

10. Cognitive Requirements of Selecting a Sample: Judgment Which records Rs will select to report in survey. How to apply instructions to their records. • How many records will be in my sample? • How many do I need to report for? • What if I cannot access all necessary records?

11. Results: Round 1 The 2002 form was used in this round of testing to identify shortcomings.

12. 2002 CFS Questionnaire, Item E

13. 2002 CFS Questionnaire, Item E

14. 2002 CFS Questionnaire, Item E

15. Sample Selection vs. “Audit” Some Rs thought we were asking for an audit of shipping records and suggested calling Item E an “Audit of Shipping Records” “Audit” can have a negative connotation Item E could be similar to an audit, but is in fact systematic sampling.

16. Results: Round 2 Purpose of this round was to resolve issues from Round 1 Item E revised from Round 1: • Placed on one page • Eliminated diagrams and split table into two pieces • Instructions turned into questions

17. Item E, Round 2

18. Results: Round 2 (cont’d) Having Item E on one page was helpful. The revised Item E did not adequately connect Items D and F. • Asks Rs to re-report their response to Item D • Later asked how many shipments they would report in Item F • Were able to complete Item E, but did not know why

19. Results: Round 2 (cont’d) Language continued to be an issue • “Selection rate” and “selection method” • Difficult for people with non-statistical backgrounds to comprehend Rs who reported <40 outbound shipments found Item E difficult to complete. • No sample needs to be drawn, but it was difficult to convey that info

20. Results: Round 3 Purpose was to ensure changes made were working to assist Rs in sample selection process. Several changes made from Round 2: • Added skip instruction to Item D, so Rs with <40 shipments would go straight to Item F (and skip sampling process) • Added purpose statement, so Rs knew why they were drawing a sample • Refined wording so it was less statistical

21. Results: Round 3 (cont’d) More changes from Round 2: • Process and instructions were numbered to assist navigation • An example was added, but not emphasized (indented and placed at bottom of the page), so Rs would not focus on it exclusively

22. Final Version of Item E for 2007 CFS Questionnaire

23. Final Version of Item E • Instructions contained less text and were numbered for easier navigation. • Only one table, where Rs could mark the row indicating which records they would select (“every nth outbound shipment”) • Sample selection process explained in 7 short steps. • Example was de-emphasized in order to act as verification that Rs did everything correctly.

24. Conclusion Very difficult for non-statisticians to do a statistical task, as in Item E. Language is key. Conveying relationship between Items D, E, and F was challenging. Most significant and helpful change: putting all of Item E on one page and removing the diagrams.

25. Conclusion (cont’d) Diagrams did not work because Rs did not read accompanying text. However, there are cases where diagrams can be helpful in surveys, especially when accompanying text can be read and easily understood.

26. Conclusion (cont’d) CFS is currently in the field and is performing very well, overall. Incoming calls: • Clarify sampling process • Express curiosity on why sampling is necessary This supports the fact that it is very difficult for non-statisticians to perform a statistical task.

27. Future Research Explore other surveys that require Rs to select their own samples. Perhaps topic of other surveys could explain the difficulty or ease of Rs completing task without much confusion Answer could lie in the amount of statistical knowledge Rs have, the field they work in, and how the survey explains the sample selection process.

28. Thank you! • For questions, please contact: • La Toya Barnett Thomas • (301) 763-9588 • La.Toya.Barnett@census.gov