Jaki S. McCarthy National Agricultural Statistics Service,

1. Like – But Oh, How Different: The effect of different questionnaire formats in the 2005 Census of Agriculture Content Test. Jaki S. McCarthy National Agricultural Statistics Service, United States Department of Agriculture

2. Census of Agriculture conducted every 5 years • Includes all known operations with agriculture potential of any type (~3mil) • Information collected on acreage, production, sales and expenses, production practices, and demographics • Primarily mail out/mail back form, with follow up mailings, limited CATI and interview follow up • Reporting is mandatory

3. Prior to Census, Census Content Test is Conducted • Test used to finalize forms, test data collection and processing procedures • Small scale mail out/mail back with follow up reminder • Follow up cognitive interviews conducted with subset of respondents • Reporting was not mandatory

4. 2005 Census Content Test (CCT) • Two forms tested (long and short) • In 2007 Census, short form will be targeted to specific types of operations • For CCT, split sample of all farms used for analysis

5. Both CCT Forms collected the same information • Differences in: • Format • Layout • Number of pages

6. Why use a short form? • May increase response rates • May be faster and/or easier to complete • Costs less to print and mail

7. Why use a long form? • More specific examples and instructions can be included • Explicit screening questions and routing instructions can be incorporated • Common items are pre-printed • Flexibility with layout can allow related items to be placed together

8. LONG FORM 24 pages Acres operated by land type collected immediately after acres operated by ownership Separate sections for different commodities; Most common items pre-printed, others listed Sales collected separately from production in list Government programs listed SHORT FORM 12 pages Acres operated by land type collected several pages after acres operated by ownership All crops and livestock collected in open write-in tables; Only a few items included as examples Value of sales collected with production in open tables Government program info collected in abbrev. tables Qs on production contracts and irrigated pasture omitted Differences Between Forms

9. Short Form, Field Crops and Hay Table

10. Long Form, Field Crop Section

11. Short and Long Forms • Split sample used to send short and long forms to all types of operations • Response rate for each form tracked • Content was the same, so comparisons of length were not confounded with content • Data compared across forms • Information from cognitive interviews compared across forms

12. Less pages ≠ Higher response *short form response weighted for comparability to long form subgroup sizes

13. Long Form: page 2 Short Form: page 2 Long Form: Facing on Page 3 Short Form: 3 pages later on page 5

14. If you want people to report the same number twice, don’t ask them 3 pages apart Problems reporting Total Acres

15. If you want bushels, ask for bushels Quantity Unit preprinted

16. Quantity Unit must be written by respondent

17. Short Form Long Form

18. If you want something reported, ask for it explicitly

19. Other problems • Short Form: Fewer gov. payments reported • Short Form: More manual processing required (extra sheets, uncodable responses, more than one number entered in a cell) • Short Form: Slightly more omitted agriculture reported in cognitive interviews (9.9% vs 7.4%) • Long Form: Self reported completion times slightly longer (44.46 vs 47.87 minutes)

20. Changes based on these findings • Added additional types of operations to exclude from short form (e.g. horses, specialties, etc.) • Use of more long form question formats on short form (e.g. government payments) • Use of additional edit checks on short form

21. Conclusions • Decreasing # of pages does not increase response rate • More data problems with the short form, particularly underreporting • Forms may collect the same data, but their performance is NOT equivalent— Like, but different

22. Plans for use of Short Form • Short Form will be used ONLY with limited set of operations (e.g. operations known to have 4 or less commodities; operations without specialty commodities; operations not previously reporting horses, goats, sheep, bees, etc) • With over 3 million mailing addresses, cost savings for lighter mailing can be substantial

23. YES, it was the mountain Echo, Solitary, clear, profound, Answering to the shouting Cuckoo, Giving to her sound for sound! Unsolicited reply To a babbling wanderer sent; Like her ordinary cry, Like--but oh, how different! --W. Wordsworth