An Application of Selective Editing to the US Census Bureau Trade Data

1. An Application of Selective Editing to the US Census Bureau Trade Data Maria Garcia US Census Bureau UNECE/SDE, Oslo, Norway, 24-26 September 2012

2. Foreign Trade Statistics Programs • Official source of US international merchandise trade statistics • Electronic data collection • Complete enumeration • Pre – editing: check for fatal errors • Micro editing • Range and ratio edits • Automatic imputation • “Rejects” – imputation not successful

3. Foreign Trade Data Processing • Rejects • Distribute among analysts for manual correction. • Analysts review large number of records under tight time constraints • Goal: Use selective editing to identify highly suspicious errors having a high potential effect on the estimates • Value (V) • Quantity (Q) • Shipping weight (SW)

4. Hidiroglou-Berthelot Method (HB) • Latouche and Berthelot (1992) used HB when developing their score functions. • HB uses historical ratios () to detect outlying observations in periodic data (1986). • Our data: cannot test historical ratios. • For record , instead of using HB to identify errors in or , identify errors in unit prices

5. HB Method for Our Trade Data • Apply a series of transformations: • Identify outliers at both ends of the distribution of unit prices • Size transformation , where

6. HB for Our Trade Data (Cont’d) • Measure distance of quartiles of from median • Measure displacement from median weighted by appropriate distance

7. Effect on Publication Totals • Examine effect of changes on final publication totals (Adapted from Latouche and Berthelot, 1992) • If no anticipated value is available, use median of ratios and reported data, e.g., for Value • For every record (Similar to Jäder and Norberg, 2005)

8. Simulation and Evaluation • Simulation • Extract from a one-year exports data file • Archived raw and edited (final) data • Evaluation • Absolute Pseudo-bias = (Latouche and Berthelot, 1992)

9. Evaluation Results • Examining results at lowest level of aggregation: • Data users may closely scrutinize the statistics for particular types of products • Ex: import/export of rough diamonds • Kimberley Process - joint governments, industry and civil society initiative to stem the flow of conflict diamonds

10. Evaluation Results

11. Evaluation Results

12. Customer’s Feedback • Subject matter experts questioned: • High ranking given to records that by experience they consider insignificant to final cell estimates • Low ranking given to records that would have been flagged for manual correction

13. Customer’s Feedback Commodity XXXXXXXXXX Suspicious record is correctly identified by selective editing as having a large effect on the total quantity:

14. Customer’s Feedback Commodity YYYYYYYYYY

15. Concluding Remarks • Combination of Hidiroglou-Berthelot and Latouche-Berthelot methods. • Tried alternative ways to calculate : Quartile method and Resistant fences method. • Looking at alternative evaluation methods, determining optimal levels of aggregation, and including seasonality in calculation of simple statistics.

16. Thank you!