Introduction to Census Archiving Session 3

1. Introduction to Census ArchivingSession 3

2. What is data archiving? Data archiving refers to the long-term storage of electronic data andits affiliated documentation (data collection instruments, study methods, descriptive information for variables and files) Archiving is not merely making copies of data Archiving/preservation is keeping digital data alive and accessible for long-term use Involves adding value to and enhancing the usefulness of the data for secondary analysis Therefore, archiving should be planned and undertaken with view point of potential long-term uses and users of the data

3. Why archive the data? A: Responsibility of data producer NSOs generate vast amounts of data which represent significant investment by country and have considerable value for present and future data uses But, data dissemination (hence utilization) generally weakest part of census operation Dissemination of micro-data even more limited Data producers have responsibility not only to collect and disseminate data but also to ensure their long-term usability through proper archiving and preservation Archiving is not just safe-keeping of data, but to promote and disseminate statistics as widely and effectively as possible

4. Why archive the data? B: Need to preserve the data for long-term usage Without proper archiving valuable data could be lost over time due to loss, obsolescence of technology, or irreversible damage Note: challenges of maintaining digital content over successive generations of technology Archiving makes it possible for researchers to conduct secondary analyses on the data and extension of original findings

5. What does archiving involve? Archiving encompasses a broad range of activities beyond data collection - it covers the entire data lifecycle Data archiving is a process and not an end state Important, therefore, to develop a plan Start process early (before data collection) and include preservation of accurate metadata in the plan For the long-term, require formulating an archiving and preservation policy as part of overall strategy for data sharing and utilization

6. Core attributes for a data archiving programme An institutional strategy for archiving should encompass: Organizational infrastructure: defines how much data will be archived (and shared) Technological infrastructure: defines how the data will be archived Resources: defines how much it will cost to archive the data Core attributes should be balanced and work in concert in order for the program to succeed (e.g., archiving is not just IT-related but needs also proper organizational arrangements and adequate resources)

7. Core attributes for a data archiving programme (contd.) • Understand process and what’s required in terms of steps for developing an archiving system • Strategy should be developed taking into account needs and requirements of individual data producers but complying with standards of good practice within the digital preservation community

8. Organizational infrastructure Archiving programme exists within an organizational context and as such must fit the needs, priorities, and resources of that organization Strong, well planned and sustainable organizational setting is crucial for long-term success of archiving programme Policies and procedures Policies reflect commitments and decisions regarding data archiving and sharing Mission statement of NSO that encompasses data archiving and sharing Legal frame work defines mandate and may cover data archiving and documentation, sharing and dissemination, particularly of micro-data Development of clear policy and strategy that explicitly address archiving commitments and decisions are important for success Written policy and procedures help to define responsibilities and requirements for archiving, sharing and access

9. Technological infrastructure Technology (technological and procedural suitability and system security) is a means to achieving archiving objectives, i.e., it’s an enabler Equipment Software Hardware Secure environment Staffing Technology enables organization to meet defined requirements for archiving – should be suitable for the purpose Decisions on technological infrastructure center on whether to build, extend, collaborate, or out-source

10. Technological infrastructure (contd.) Technology is not static - organization to actively monitor technological developments and systematically consider potential implications for archiving programme (software and hardware) The infrastructure should plan for technological changes and their attendant costs As digital storage media (formats and physical storage) media will ultimately become absolete: store multiple copies on different storage media and different file formats Need to develop appropriate data migration procedures Important to maintain a flexible preservation system that evolves to meet demands of changing technology and new and increasing user expectations

11. Resource framework Organizational and technological infrastructures are not sustainable without an on-going commitment of resources Organization should identify appropriate resources to develop and maintain archiving programme Resources should cover start-up, on-going and contingency funding Sustainable funding should be designated for cost of staff, equipment, software, and storage costs - to demonstrate the organization’s commitment to programme

12. Major activities in archiving of micro-data Documentation Annonymization Dissemination Discussed in brief with details provided in subsequent sessions

13. Role of documentation Documentation explains how the data were collected, their content and structure and any manipulation that may have taken place Documentation is required in order to understand and interpret the data by providing a context Without proper documentation, data are useless Some items for documentation: Context of data collection Data collection methods Data validation, cleaning and quality assurance procedures Descriptions of variables Definitions of codes Definitions of scientific terminology Treatment of missing values Also allows reuse of documents for future surveys Metadata categorized according to international standards (ISO, DDI, Dublin Core)

14. Anonymization Dissemination of micro-data posses challenges for protection of data confidentiality Appropriate procedures should be developed to measure disclosure risks Develop appropriate procedures for anonymization of data so as to reduce risk of disclosure of confidential information

15. Dissemination Benefits of disseminating micro-data (broadening use of existing data and increasing return on data collection investments) BUT there are also associated costs and risks (financial cost, exposure to criticism, disclosure of confidential information) Benefits have to be weighed against ethical and legal obligations to keep respondent information confidential Design dissemination strategy to provide access to micro-data while implementing procedures to guard against breach of data confidentiality Dissemination strategy should include determination of types of outputs, access methods, procedures for terms of use of the data, as well as possible penalties for breaching terms of use, etc.

16. Challenges to implementation of an archiving programme Resource constraints Lack of specific laws/regulations and policies for archiving Lack of institutional commitment Inadequate IT infrastructure Lack of appropriate skilled personnel Implementing data anonymization procedures Lost data in original census databases No general guidelines for choosing a suitable platform to archive census data Lack of mechanism to cross check the quality of archived micro-data Integration and harmonization of data from successive censuses

17. Thank You!