Comparing ACS with Census Sample and Current Survey Data

1. Comparing ACS with Census Sample and Current Survey Data Partnership and Data Services Training Susan Love April 12, 2005

2. Some Sources for Comparison • Census long form samples in general and Census 2000 in particular • Current Population Survey and Annual Social and Economic Supplement • American Housing Survey

3. What should we compare? • Sampling frames and sample design • Data collection time frames and modes • Interviewer requirements • Respondent requirements • Data completeness • Weighting and estimation methods • Final results at comparable geographies

4. ACS and the Decennial Census

5. How do the samples compare? • Similarities • Systematic samples of MAF addresses • Differential sampling rates for areas based on population size

6. Sampling Rates

7. How do the samples compare? • Differences • ACS samples every year and spreads sample over 12 months; census samples once a decade and uses the entire sample at the same time • ACS subsamples for personal visit followup; court ruled against census subsampling • Census sample estimates based on about 18 million housing units; ACS 5 year estimates based on about 11 million housing units

8. How does data collection compare? • Similarities • Primary method is self-response to a paper questionnaire • Content is the same

9. How does data collection compare? • Differences • ACS nonresponse FU uses CATI and CAPI instruments; past censuses have used only paper questionnaires • ACS data collected only from household members; census data often collected from neighbors • ACS interviews conducted by experienced and well-supervised staff; census enumerations conducted by inexperienced temporary workforce

10. What is the impact on the data? • ACS estimates have higher sampling error • measures are released for each estimate and shown as 90% confidence limits or margins of error in every table • Similar sampling error measures for census long form sample estimates have not been provided

11. What is the impact on the data? • ACS estimates have lower potential nonsampling error • Unit nonresponse: no data for a case • Item nonresponse: data missing for an item • These and other measures are released on ACS’s Quality Measures website • www.census.gov/acs/www/UseData/sse/index.htm

14. What do the estimates represent? • Most ACS estimates are said to be the average characteristics of an area over a calendar year • Census sample estimates are often said to be characteristics of an area as of Census Day • Is this so?

15. What do the estimates represent? • Both the ACS interview date and the census enumeration date play important roles • ACS uses the interview date as the single reference point, or as the end of a reference period, for all data collection • The Census 2000 sample did too, except for income, migration, and agricultural sales

16. Weighted Census 2000 Sample and 2000 ACS Housing Units by Response Week

17. What do the Census 2000 sample estimates represent? • Census 2000 data collection was a sequential process of 3 operations over a 6 month period • Data from these operations were processed together, weighted and tabulated • The characteristic estimates are the sum of the peaks and valleys of the enumerations, not Census Day

18. Census 2000 Data Collection Design

19. What do the ACS estimates represent? • ACS data collection is a sequential process of 3 operations on each of 12 monthly samples over 3 month periods • All 3 operations take place every month on different sample panels, a continuous series of interviews • Data collected in a calendar year are grouped by interview month; a weighting adjustment is applied to smooth overall monthly differences • Summed estimates are considered the average characteristics of all areas for the calendar year

20. ACS Data Collection Design

21. How did the 2000 ACS and Census 2000 results compare?

22. Comparisons with Census 2000 • Four reports by Bureau staff compared the 2000 ACS data with Census 2000 data at the national level • Two staff reports and four reports by outside researchers compared three-year averages from the 1999-2001 ACS test sites with Census 2000 at county and tract levels www.census.gov/acs/www/advMeth/Reports

23. Comparisons with Census 2000 • ACS profile distributions were compared to Census 2000 profile distributions without group quarters pop • ACS distributions were surprisingly similar to Census 2000 • Only about 9% of the profile table estimates differed by more than 1 percentage point • Self-response data from mail returns were responsible for the consistent results

24. Comparisons with Census 2000 • Characteristics differing the most • Race and relationship (Table 1) • Ancestry, disability, high school graduates (Table 2) • Labor force participation, median household and family income, and poverty rates (Table 3) • Year built, rooms, and house heating fuel (Table 4)

25. Summary of National-Level Comparisons of General Demographic and Housing Characteristics (Table 1)

26. General Demographic Characteristics • ACS estimated more White and fewer Some Other Race – experience and training of FRs • ACS estimated fewer married couple families – ACS weighting, edit difference (marital status no longer 100% census item) • Item nonresponse – Race: Census 3.9%, ACS 2.4%. Relationship: Census 2.2%, ACS relationship 1.6%

27. Summary of National-Level Comparisons of Social Characteristics (Table 2)

28. Social Characteristics • ACS estimated more English, German, and Irish ancestry -- supplementary sample design • ACS estimated more people with high school diploma and no college – time frame, data capture • ACS estimated fewer people with disabilities – poor question design, answered well only in CATI and CAPI • Item nonresponse – Ancestry: Census 19%, ACS 11.7%. Ed. attain.: Census 7.2%, ACS 4.8%. Disability: Census 8.5%, ACS 2.1%

29. Summary of National-Level Comparisons of Economic Characteristics (Table 3)

30. Economic Characteristics • ACS estimated higher labor force participation – slight question and edit difference • ACS estimated more families with young children and all parents in the labor force – weighting and edit differences • ACS estimates a lower median household and family income and higher poverty – data capture • Item nonresponse-- Labor force status: Census 11.1%, ACS 6.0%. Household income: Census 30%, ACS 24%.

31. Summary of National-Level Comparisons of Housing Characteristics (Table 4)

32. Physical Housing Characteristics • ACS estimated more units built in 1939 or before – troublesome question (multi-unit rentals) • ACS estimated more 4 and 5 room units – troublesome question; old edit assumptions • ACS estimated fewer units heated with gas – troublesome question (multi-unit rentals) • Item nonresponse -- Year built: Census 12.7%, ACS 14.9%. Rooms: Census 7.8%, ACS 4.2%. Heating fuel: Census 7.4%, ACS 2.1%

33. Financial Housing Characteristics • ACS estimated fewer owner-occupied units with a mortgage – edit difference (more info in ACS) • ACS estimated fewer households at the low end and more households at the high end of gross rent as a % of income – higher income in Census 2000 than in ACS (and ASEC) • Item nonresponse– First mortgage: Census 6.0%, ACS 2.0%. Gross rent: Census 37.7%, ACS 20.3%

34. Summary of Reasons for Differences • Timing and reference periods • Quality of nonresponse followup • Training and staff experience • Treatment of proxy information • Data completeness • Troublesome questions • Weighting

35. Conclusions • Most estimates are very comparable, and the differences are not unexpected or unexplainable • ACS has higher level of overall response and individual item response, so less chance of nonresponse bias • ACS household information comes only from a member so probably more accurate • ACS is a better way to collect this wide-ranging information than was the decennial census because the distribution of the data over the collection time frame is more meaningful

36. What about current surveys?

37. Comparisons with CPS & ASEC • Coming this year -- • Industry, Occupation, and Class of Worker • Income and Poverty • Race and Ethnicity • Households and Families • Age and Sex

38. Comparisons with CPS and ASEC • 73,000 addresses in CPS sample and 250,000 address in ACS sample each month; 100,000 address in ASEC each year • CPS interviews about 55,000 households and ACS interviews about 170,000 households each month • CPS units interviewed 8 times in 2 years; ACS and ASEC units interviewed only once • ASEC interviews nearly 78,000 households once a year; ACS interviews about 2 million households over the course of a year

39. Comparisons with CPS and ASEC • Monthly CPS collects detailed labor force participation and releases the official estimates every month • ASEC covers several topics, releases national and state data yearly, and produces the official national income and poverty estimates • ACS produces data on many of the same topics and releases data for the nation, states, and lower-level geographies with populations of at least 65,000 every year

40. Comparisons with CPS and ASEC • CPS and ASEC focus on fewer topics and go into them in greater depth than ACS • They produce more complete measurements of concepts; ACS only touches on many topics, as did the census long form samples • ACS produces information for much lower levels of geography because of its larger sample

41. Comparisons with American Housing Survey (AHS) • Coming next year -- • Measuring the financial characteristics of housing • Homeownership • Property value • Selected monthly owner costs • Contract and gross rents

42. Comparisons with the American Housing Survey • About 64,000 addresses in AHS national sample, with about 55,000 interviews conducted every 2 years; ACS samples nearly 3 million addresses every year, resulting in over 2 million interviews • AHS has visited many of the same sample addresses every 2 years since 1985; ACS visits a different sample every month

43. Comparisons with the American Housing Survey • AHS collects very detailed information on physical and economic characteristics and conditions of housing and neighborhoods; releases data at national and regional level, and for 6 large metropolitan areas • ACS produces data on some of the same topics but not in detail; releases information at much lower levels of geography

44. General Guidelines for Single-year Annual Estimates • Use ASEC for national estimates of income and poverty, and ACS for lower levels of geography • Use CPS for national and state estimates of basic labor force participation, and ACS for estimates by socioeconomic characteristics at lower levels of geography • Use AHS for detailed characteristics of housing at the national level and 6 large metro areas, and ACS for lower levels of geography.

46. ACS Data Products • In 2005 for 2004 ACS • Redesigned Base (detail) table package – over 900 • New Subject tables – about 50 • New Selected Population Profiles – well over half of the SF-4 groups • Redesigned and expanded tabular profiles • Expanded narrative profiles • In 2006 for 2005 ACS • Further narrative profile expansion • New thematic maps

47. Illustration of an ACS Release schedule