Reengineering SIPP

1. Reengineering SIPP Constance F. Citro, Director Committee on National Statistics ASA/SRM SIPP–November 17, 2009

2. CNSTAT Reports on SIPP 1993 – Comprehensive review 2009 – Focus on administra- tive records; Chaired by Karl Scholz; John Czajka wrote data quality appendix ; Joe Hotz on BOTH panels

3. Panel Members John Karl Scholz (chair) John Iceland F. Jay Breidt Jerome P. Reiter Leonard E. Burman Deanna T. Schexnayder John L. Czajka Robert F. Schoeni Peter Gottschalk Jennifer Van Hook Ronald T. Haskins Pat Ruggles (study V. Joseph Hotz director thru 6/2008)

4. Charge to the Panel • Consider the advantages and disadvantages of strategies for linking administrative records and survey data . . . . • Consider alternative uses of administrative records . . . that do not require actual data linking (e.g., to evaluate SIPP data quality). • Consider interview periodicity, mode of data collection, and sample source and size.

5. SIPP’s Unique Contribution • Monthly data for analysis of intrayear transitions in marital status, poverty, employment, health insurance coverage, and eligibility for and participation in a wide range of government assistance programs • As vital today as when SIPP began in 1984 • Reengineering must foster SIPP’s ability to measure short-term dynamics with improved quality and timeliness

6. Administrative Records—SIPP Needs Them Conclusion 3-1 Administrative records . . . cannot replace SIPP, primarily because they do not provide information on people who are eligible for—but do not participate in—government assistance programs . . . . Administrative records can and should be used in a reengineered SIPP [to improve quality].

7. History SIPP has a long history of using administrative records, dating back to the Income Survey Development Program in the late 1970s • sample frames • exact matches • forward and reverse record checks • aggregate comparisons with benchmarks • poststratification weight adjustments

8. BUT—historical usesnot cumulative SIPP’s uses of administrative records have been sporadic, often limited one-shot efforts What is different now? Two factors: • Increasingly important to use records to improve quality of income and program data • Census Bureau has infrastructure on which to build (before, administrative records were always the wave of the future)

9. Importance – Deteriorationin Income Reporting • From 1993 to 2002, SIPP income for the bottom quintile (its strength) decreased from 120 to 106% of CPS ASEC income • SIPP TANF income was 73% of benchmark in 1987, rising to 77% in 1996, and dropping to 62% in 2002 (CPS ASEC worse) • Percent of income imputed increased from 21% in 1993 to 29% in 2002 (CPS ASEC worse)

10. Infrastructure • From work begun in the 1990s, Census Bureau has a fully matched and unduplicated E-StARS file, updated annually (heart is IRS and Medicare records) • Bureau regularly acquires other federal files (e.g., SSA and Medicaid), plus state employment/earnings files (LEHD program) • Bureau has world experts in matching (no longer need SSNs)

11. Framework—Potential Benefits • Improved quality of income and program participation data • Can’t assume admin records are “gold standard,” except when they are (e.g., benefits paid out versus characteristics of beneficiaries) • Additional data not easily obtainable otherwise (e.g., longitudinal earnings records in SIPP gold standard file)

12. Framework—Potential Costs • Resources to acquire, clean, and apply records—staff costs can be substantial just for negotiating acquisition agreements • Delays in ability to release data products (or else use outdated records with current survey data) • Need for increased confidentiality protection, which could affect data use, especially for policy analysis—SIPP users want public use files

13. Indirect Uses (records notpart of survey file) • Aggregate comparisons with benchmarks • Individual-level comparisons through exact matching • Enhancement of imputation and weighting procedures for item and household nonresponse

14. Direct Uses Using administrative values directly for item nonresponse Adjusting survey responses for under/overreporting Using administrative values and dropping the corresponding survey questions Appending administrative values to survey records

15. Trade-offs Indirect uses are advantageous because they would have little or no adverse effect on timeliness or confidentiality protection; BUT, less powerful for improving quality Direct uses potentially offer significant quality improvements but at the potential cost of delays and reduced access for data users

16. Strategic Approach • Census Bureau needs to evaluate the benefits and costs of acquiring and using different kinds of records in a reengineered SIPP and make strategic decisions about which records to acquire in the short and longer terms and which uses to put them to in the short and longer terms

17. Strategic Approach forAcquisition • Go after federal records first—e.g., veterans benefits; state laws/practices/organizational structure vary greatly (Sylvester paper) • Go after income/program types that are feasible to acquire and most important for low-income population, particularly during economic downturns— UI benefits a top priority

18. Strategic Approach forUse – Indirect First • Regular aggregate comparisons are easy and valuable—data quality has varied over time, so trends are important to follow • Exact matches, where feasible, are also valuable in prioritizing acquisition and use • Program rules and records can help improve imputations, which should be model-based

19. Implementation and Concurrent R&D While moving ahead to implement indirect uses, need to proceed with R&D program for potentially even more valuable direct uses OASDI and SSI benefits are prime candidates for adjusting survey responses or replacing survey questions (records are timely) Also move ahead on partial synthesis methods for protecting confidentiality

20. Don’t Go It Alone; Enlist Allies Recommendation 3-5 The Census Bureau should request [OMB] to establish an interagency working group on uses of administrative records in SIPP. The group would include technical staff from relevant agencies who have deep knowledge of assistance programs and income sources along with Census Bureau SIPP staff . . . .

21. Use Total Error Framework for Survey and Ad Records We encourage the Census Bureau to view the errors in administrative records and in matches of them with survey records in the same manner that … statistical agencies have commonly viewed nonresponse and reporting errors in surveys—namely, as problems to address but not a brick wall. (p. 67)

22. Innovation in Design andData Collection • Caution needed on Event History Calendar with annual interviews, given that it affects SIPP’s most important feature—monthly data • There needs to be adequate evaluation of EHC and also a bridge to new design • Recommendations 4-2, 4-3: Overlap traditional and EHC SIPP panels for 2 years, beginning in 2012

23. Expanded/New ContentAdvisory Group • Assistance programs change and so SIPP content will need to change—advisory group should have input • Census Bureau administers SIPP in trust for its research and policy users, so needs more than usual outreach and involvement of them (has begun well with “content matrix” meetings)

24. Interaction of EHC andAdministrative Records • Clearly, cost a prime motivator to moving to EHC with annual interviews; while cost-effectiveness is obviously important, preserving and enhancing SIPP’s unique contribution of monthly data is paramount • EHC likely to have strengths and weaknesses; if EHC/annual interviews adopted, Census should be prepared to make direct use of administrative records to shore up weak areas

25. Conclusion • SIPP can greatly benefit from administrative records, although extensive use of records won’t happen overnight; EHC/annual interviews not likely a magic, one-time fix • Sustained, cumulative R&D program is needed; strategically planned to foster SIPP’s primary goal to provide data on intrayear dynamics of employment, income, and program eligibility and participation