Download
1 / 35
350 likes | 500 Views
IPUMS-International Integration Process. Matt Sobek Minnesota Population Center sobek@pop.umn.edu. 1. 2. 3. 4. Input material. Pre-processing. Standardization. Integration. Data files. Batch samples Reformat data Donation Draw sample Confidentiality A. Code clean-up Verify data
E N D
IPUMS-InternationalIntegration Process Matt SobekMinnesota Population Centersobek@pop.umn.edu
1 2 3 4 Input material Pre-processing Standardization Integration Data files Batch samples Reformat data Donation Draw sample Confidentiality A Code clean-up Verify data Confidentiality B Harmonize codes Variable programming Constructed variables Data dictionary Enumeration forms Enum. instructions Sample information Translate to English Images to editable files Ipums data dictionary Tag enumeration text Document unharmonizedvariables Variable descriptions Sample design
End Matt SobekMinnesota Population Centersobek@pop.umn.edu
Batch Samples • In spring we identify the samples to integrate the following year. • Samples are processed as a group – one per year. The entire batch of samples is processed through each stage before we proceed to the next step. • There is little flexibility in the work process. If a sample is not available for processing during the earliest stages of integration, it cannot be included in the data release for that year.
Original Input Data • Some examples of differing file formats: • SPSS and SAS system files • Redatam-format • IMPS format • Records that combine household and person characteristics • Separate files for persons, households (and dwellings, buildings) • Different types of records (mortality or migration) • Separate files for different administrative units
Reformatting: Rectangular Sample (Person records only; household data duplicated on person records) (Brazil 1980)
Reformatting: Dwelling-Household-Person Sample (Separate dwelling and household records) (Chile 1992)
Reformatting: Merge Household and Person Files Household File Person File (Brazil 2000)
Reformatting: Persons not Organized in Households (Individuals only; not organized in households) (Mexico 1960)
Donation and Error Correction • Data are tested for errors that affect structural integrity, such as merged households, unmatched person and household records, corrupted records, etc. Such errors often do not affect tabulations, but create inconsistencies across records within households that affect sophisticated analyses. • Some problems can be resolved with custom programming. • Other problems are resolved by donating (substituting) a donor household for the corrupted one. • Households are divided into strata based on predictor variables. Donors are drawn from the same strata as the corrupted household, ensuring they share key characteristics. • If a sample is drawn from the full census, a substitute donor record is used; if we are already starting with a sample, the donor record is duplicated. A flag indicates that a record was duplicated.
Drawing a Sample About one-third of IPUMS samples are drawn from full-count data. After reformatting, we draw a systematic sample of every Nth dwelling to yield the desired sample density – typically 10%. If the input data are not full-count (for example, they include only the long-form records), the sample design might have to account for differing sample densities between areas. Very large dwelling units (over 30 persons) are sampled at the individual level – not as intact units – in order to reduce sampling error. Every Nth individual is taken.
Confidentiality Measures: A Swap a small percentage of cases between geographic areas. Reorder households within geographic areas. Suppress low-level geographic variables. Suppress any variable deemed too sensitive by the National Statistical Office. Encrypt all versions of the data prior to the imposition of these confidentiality measures.
Code Clean-Up: Recoding Unharmonized Variables • Recode the input variables to conform to some basic standards for treatment of missing values, etc. • Recode stray values into a consolidated missing category as appropriate. • Convert non-numeric characters to numeric. • Most recoding is performed using a data translation matrix like the one below for Marital Status in 1984 Costa Rica. If the recoding requires more complex logic, use custom programming.
Verify Data: Unharmonized Variables Examine the marginal frequencies of every input variable. Analyze the data universe for each variable – the population at risk of having a response. Determine the theoretical universe from enumeration materials or other documentation, then empirically determine any discrepancies from that universe. Document the universe for each variable and any other observations.
Confidentiality Measures: B • Recode geographic units to ensure small localities cannot be identified (typically those with fewer than 20,000 persons). • For recent censuses: • Identify cells that represent very small numbers of persons in the population. Code them to a residual category or combine them. • Top- or bottom-code continuous variables that have a long tail that could identify small subpopulations. • Suppress specific categories of variables as requested by the National Statistical Office.
Harmonize Codes: Translation Matrix for Marital Status China 1982 Colombia 1973 Kenya 1989 Mexico 1970 U.S.A. 1990
Variable Programming Some variable manipulations are too complex to be handled using the translation matrix tables. Typically these involve continuous variables or recoding logic that refers to multiple variables. This programming is written in C++.
Constructed “Pointer” Variables (Simple household) Spouse’s 2 1 0 0 0 0 Mother’s Father’s 0 0 0 0 0 0 2 1 2 1 2 1 (Colombia 1985)
1 1 1 1 Constructed “Pointer” Variables (Complex household) Spouse’s Mother’s Father’s 0 0 0 0 0 0 0 0 0 6 0 5 0 0 0 5 6 0 5 6 0 0 0 0 9 0 0 9 0 (Colombia 1985)
Sample Information – from Statistical Office Sample information is difficult for the IPUMS project to collect. Often only limited information can be gleaned from available documentation. It is extremely helpful when countries collate the information themselves, as was done below by the Netherlands:
Translate Documents to English Many countries provide their census documentation in English. For those that do not, the IPUMS project hires translators from around the world. Often these are persons currently or formerly associated with National Statistical Offices. Some common languages are translated by staff in Minnesota.
Document Unharmonized Variables The enumeration form and instruction text provides most of the documentation for the unharmonized input variables. Other documentation is written as needed to clarify the interpretation of the variable for users. We also empirically determine the universe of persons or households with valid values for each variable.
