Why Should the Federal Government Care?

1. Why Should the Federal Government Care? Clyde Tucker Bureau of Labor Statistics

2. Reasons Not to Care • Getting the numbers out on time is how we’re judged • Quality is fine but it’s of secondary importance • Nobody can know the truth anyway

3. Destroying Myths • Obviously, a lot of people are interested in quality • Actually, they want everything—speed and accuracy • Maybe nobody knows the right answer, but everyone thinks they do

4. Problems to Overcome • Skill level of the Federal workforce • Multiple measures with different kinds of errors • Skepticism of program managers • Practical concerns • Time lag • Method of reporting

5. Skills of the Workforce • New field and rare skill • Hard to get and keep talent • Focus of much of research on production issues (response rates, question wording, imputation, variance, etc.) • Advances in field likely to come from outside

6. Multiple Measures With Different Errors • Programs have multiple outputs • There are multiple programs • Broad range of error properties and methods of analysis • Agreement on where to start can be difficult

7. Skepticism of Program Managers • Haven’t given them much up to now • Placing faith in a somewhat incoherent field—nonsampling error research • Don’t you think they would report the right answers if they had them • Will we do more harm than good—subjecting programs to greater criticism • Can the measures of nonsampling error be trusted (What are their error properties?) • What’s it going to cost and is it worth it—cost benefit analysis needed • Even if cost-effective , there still may not be the money • Even if we had good measures of these errors, can we use them to improve methods (e.g., continuous improvement)

8. Practical Concerns • Time lag • Unless the measures of nonsampling errors, like variances, are produced at the same time as the estimates they will not be very useful • Method of reporting • Will MSE now be used? • What about the asymmetrical nature of bias? • What exactly are we talking about, anyway—if we knew the right answer we’d report it and not a measure of error • We’re likely to have different ways of measuring different errors, not just one, like variance • Do these become our performance measures?

9. So Is the Federal Government Doing Something Already • Agencies now have information quality guidelines specifying procedures for addressing complaints about the quality of estimates • OMB shortly will be releasing new survey standards, including guidance on nonresponse bias • Some agencies already have their own standards • Some surveys have a discussion of nonsampling errors in their documentation (e.g., Chapter 16 of CPS Technical Paper 63) • The FCSM interagency subcommittees on both household and establishment survey nonresponse have workgroups on nonresponse bias

10. What the Federal Government Is Doing Continued • Research • Hansen, et al. (1951, 1964, 1967) • Bailar (1983) • Linacre and Trewin (1989, 1993) • Groves and Couper (1998) • Other Census match studies • IRS and SSA studies

11. Research on BLS Surveys • BLS Consumer Expenditure Survey --Silberstein (1989); Tucker (1992); Kojetin and Jerstad (1997); Schober and Conrad (1997); Rips, et al. (2001);Tucker, et al. (2003) • Current Population Survey– Martin, Campanelli, and Fay (1991); Cohany, Polivka, and Rothgeb (1994); Polivka and Miller (1998); Kojetin and Mullin (1995); Biemer and Bushery (2000); Dixon (2004); section of Survey Methodology (December 2004) • Current Employment Statistics Program—Phipps and Tupek (1991); Copeland (2004); Tucker (2005) • American Time Use Survey—Fricker (2005?)

12. Looking to the Future • Development of expertise in the field • Long-term commitment of resources to developing measures of nonsampling error • Collaborating with those outside government • Figure out how to report errors and use them • Evaluate the quality of the measures of nonsampling error • Conduct cost-benefit analyses of their value