Discussion: Managing Small Domain Estimation Applications in the Context of General Survey Design

1. Discussion: Managing Small Domain Estimation Applications in the Context of General Survey Design John L. Eltinge U.S. Bureau of Labor Statistics Discussion for COPAFS/FCSM Session #6 December 4, 2012

2. Acknowledgements and Disclaimer The author thanks David Banks, Paul Biemer, Moon Jung Cho, Larry Cox, Don Dillman, Bob Fay, Jeffrey Gonzalez, Brian Harris-Kojetin, Rachel Harter, Mike Hidiroglou, Anders Holmberg, Bill Iwig, Alan Karr, Sallie Keller, Boris Lorenc, Bill Mockovak, Sally Morton, Barbara O’Hare, Polly Phipps and Clyde Tucker for many helpful discussions of the topics considered in this paper. The views expressed here are those of the author and do not necessarily reflect the policies of the U.S. Bureau of Labor Statistics.

3. Overview Many thanks to the presenters for three very interesting papers. All three papers consider issues important for management and policy decisions regarding small domain estimation. I. Four Components of General Survey Design that are Important for Management of Small Domain Estimation II. Liu et al. III. Rumcheva et al. IV. Joyce et al. V. Closing Remarks

4. I. Management of Small Domain Estimation through Four Components of Survey Design A. Large-Scale Statistical Production 1. Labor intensive (largely data collection) 2. Capital intensive (largely intangible capital) B. Small Domain Estimation 1. Stakeholder requests: estimates for many domains 2. Standard direct estimation: prohibitively large sample sizes 3. Instead, attempt to meet needs in (1) by replacement of some labor with capital (auxiliary data, modeling, production systems)

5. I. Four Components of Survey Design (Continued) C. For Management and Policy Decisions on Small Domain Estimation, Consider Four Components of General Survey Design: 1. Market Definition 2. Methodology 3. Systems 4. Organization Management Each of (1)-(4) are important for the papers in this session

6. II. Liu et al. A. Work with “local specificity” is a good illustration of the importance of Clear Market Definitions 1. Meeting specific perceived “customer needs” 2. Realistic alignment of cost/quality/risk profile with current and sustainable revenue streams 3. Transparent and relevant communication of (1)-(2) Crucial issue: Different stakeholders may have different perceptions related to random variability - Abstract statements - Realizations of specific random variables (“my town”)

7. II. Liu et al. (Continued) B. Primary Focus of Liu et al.: Local Specificity 1. Arises from crucial feature of the “statistical market” a. Many stakeholders b. Most have utility functions tuned to a relatively small number of domains (“my town”) 2. Issue: To what extent does the estimate for my town depend on data from - My town? - Somewhere else (via model)?

8. II. Liu et al. (Continued) 3. Liu et al. are responding to concern about risk of model failure (local lack of fit, change of model since previous model development work) 4. Comprehensive assessment of (3) requires multiple diagnostics 5. Liu et al. suggestion: Under conditions, a higher degree of “local specificity” is associated with reduced impact of model failure (if failure occurs) Thus, address part of (3) through a simple measure of “local specificity” based on sample size.

9. II. Liu et al. (Continued) C. Potential advantage of Liu et al. focus on low/medium/high local specificity: Simplicity and accessibility to non-statisticians D. Prospective extensions: Incorporate more complex measures: 1. Decomposition of domain-specific expected function of deviation of direct estimator from final estimator: sampling error, equation error, model lack of fit 2. Impact of specific types of model failure on estimates for particular domains (cf. influence functions)

10. III. Rumcheva et al. A. Address important issues at the intersection of market definition, methodology and organization management 1. Standard management, plus individual and institutional incentives 2. Small domain estimation: a. Complex cost structures - Data, production systems, highly skilled personnel - Often have large fixed components of cost b. Clear communication with stakeholders regarding quality of published estimates and related risks

11. III. Rumcheva et al. (Continued) B. NHIS (2010) Example: 1. For n domains and k binary outcome variables 2. Use RELV criteria to identify variables k with substantial variability in Complex cost and risk functions 3. Resource allocation decision: For which outcome variables k does vary enough across domains ito warrant investment in small domain modeling work?

12. III. Rumcheva et al. (Continued) C. Potential Extensions: 1. Supplement RELV with criteria that reflect: a. Possible skewness in distribution of true (uniform priors may not hold) b. Utility functions of key stakeholders: Ex: Magnitude of difference (cf. usual discussion of practical significance) c. Areas (a) and (b) provide good opportunities for practical work with elicitation of priors, utility functions For n domains and k binary outcome variables 2. Use RELV criteria to identify variables k with substantial variability in Complex cost and risk functions 3. Resource allocation decision: For which outcome variables k does vary enough across domains ito warrant investment in small domain modeling work?

13. III. Rumcheva et al. (Continued) 2. Expanded decisions on resource allocation a. Which outcome proportions do we consider for initial modeling work? Criteria: RELV or extensions from (C.1) b. Which outcome proportions do we select for full-scale production of small domain estimates? Additional criteria: Quality of model fit (degree of improvement in estimation and inference for ) relative to stakeholder needs

14. IV. Joyce et al. A. A good illustration of integration of highly sophisticated methodology with complex and robust production systems Methodology: 1. Historical focus of small domain work 2. Usual components: data quality, estimation, inference accounting for multiple sources of variability

15. IV. Joyce et al. (Continued) Production Systems 1. All aspects of production systems 2. Discrete and identifiable risk factors in production are often important 3. Small domain estimation: a. Integration of multiple data sources (surveys, administrative records) b. Model diagnostics: Timely, actionable

16. IV. Joyce et al. (Continued) C. For Joyce et al., also note importance of intermediate decisions based on empirical results Ex: Forming groups of similar jurisdictions

17. IV. Joyce et al. (Continued) D. As with Liu et al. and Rumcheva et al., important to provide stakeholders with transparent and accessible descriptions of: 1. Relative magnitudes of the effects of multiple sources of variability 2. Sensitivity of final results to sources in (D.1) - Not easy, but has an important effect on stakeholder acceptance, institutional credibility

18. IV. Closing Remarks A. Many thanks to the authors for three fascinating papers that highlight important issues in for practical management of small domain estimation B. Management and policy decisions on small domain estimation involve a complex balance among cost, data quality and risk C. View “management” of small domain estimation as fundamental part of general survey design: Market definition, methodology, systems, and organizational management

19. John L. EltingeAssociate CommissionerOffice of Survey Methods Researchwww.bls.gov/ore202-691-7404eltinge.john@bls.gov