Quality Metrics for Assessing the Impact of Editing and Imputation on Economic Data

1. Quality Metrics for Assessing the Impact of Editing and Imputation on Economic Data • Broderick E. Oliver • and • Katherine Jenny Thompson • Office of Statistical Methods and Research for Economic Programs

2. Outline • Motivation for the study • Quality Metrics (Formulas) • Quality Metrics (Actual Results) • Future Research

3. Motivation • Economic Directorate conducted a series of • studies to evaluate the editing efficiency of • selected surveys and censuses. • 1. What value is added from subjecting the same • record to multiple editing phases? • 2. What is the impact of editing and imputation on • the final data?

4. Development of Quality Metrics • Assess overall changes to “reported” data at the: • Micro level • Macro level • Examine • the size of change to reported data. • the source of change to reported data. • Determine which changes had greatest impact on final tabulations

5. Key Terms • Critical Item • Reported Data • Final Data • Data Flag

6. Metric 1 • Item Level (Critical Items) • Percentage of records with reported values whose value was changed by editing/imputation • Where: yi= 1 if reported value  final value • 0 otherwise. • and n = number of records

7. Metric 2 • Item Level (Critical Items). • The percentage of changes to the records with reported values that is attributable to analyst correction versus machine correction. • Where ai =1 if reported value  final value and source is analyst correction. • 0 otherwise. • mi =1 if reported value  final value and source is machine correction. • 0 otherwise • n = number of records.

8. Metric 3 • Item Level (Critical Items). • The source of change of the reported data. • The size of change of the reported data. • The impact of the changes on the final tabulations.

9. Metric 3: Tabular Format: (Item Level)

10. Metrics Applied to: • Annual Wholesale Trade Survey (AWTS) • Annual Survey of Manufactures (ASM)

11. Annual Wholesale Trade Survey(AWTS) • Sample Survey • Approximately 8,000 wholesale businesses • Critical Items: • Sales • Total Purchases • Total Inventories • Processed in Standard Economic Processing System (StEPS)

12. AWTS Editing/Imputation • StEPS Automatic Processing Flow • Simple Imputation Module: Data “clean up” • Edit Module: Identifies “suspicious” values • General Imputation module: Replaces “suspicious”values • Item Flagging • Can identify four distinct sources of change: • Analyst Correction • Analyst Impute • Machine Correction • No Change • “Cycling” between analyst and machine corrections

13. Annual Survey of Manufactures (ASM) • Sample Survey • 55,000 establishments • Critical Items: • Cost of Materials • Employment • Annual Payroll • Receipts • Processed in the Economic Census System • Plain Vanilla Editing Module

14. ASM Editing/Imputation • ASM Automatic Processing Flow • Pre-editing Module: Data filling and clean up • Plain Vanilla Edit Modules • Ratio (editing/imputation) • Balancing (editing/imputation) • Item Flagging • Can identify three sources of change: • Analyst correction/impute (cannot distinguish) • Machine impute • No change • “Cycling” between analyst and machine

15. Illustration of Metric 1: AWTS • Relatively few of the reported values for each critical item changed. • Changes to these records had a great impact on final tabulations.

16. Illustration of Metric 1: ASM • Relatively few of the reported values for each critical item changed. • Except for employment, changes to these records had a “small” impact on final tabulations

17. Illustration of Metric 2: AWTS AC = Analyst Correction; AI = Analyst Impute; MI = Machine Impute

18. Illustration of Metric 2: ASM AC = Analyst Correction MI = Machine Impute

19. Key Findings With Metric 3: AWTS • Analyst corrections accounted for the majority of the changes to all three critical items • Correction of “rounding” errors • Corrected by analysts • Most substantive impact on tabulations • Relatively few records

20. Key Findings Metric 3: ASM • A high percentage of changes to reported data fell into the “small change” categories. • For Cost of Materials, machine imputes made the majority of these small changes (74.7 percent). • For Receipts, analysts made the majority of these changes (68.4 percent). • Correction of “rounding” errors: • Corrected equally by analyst and machine • Most substantive impact on tabulations • Relatively few records

21. Study Highlights/Key Findings • Importance of rounding errors: • Small number of cases • Resolved generally by analysts in AWTS • Resolved by analysts and machine in ASM • Large proportion of small changes in ASM: • Identified potential edit parameter problems

22. Advantages of Standardized Metrics • Allowed for direct comparisons between different programs. • Uncovered different areas of investigation in different programs. • Facilitated “buy-in” from all parties via development process. • Provides baseline measures for future investigation.

23. Future Research • Apply metrics at various processing stages (AWTS). • Apply metrics at industry level. • Examine the number of times the records are subjected to changes.

24. Contact Information • broderick.e.oliver@census.gov • katherine.j.thompson@census.gov