Documentation and Cataloguing in Data Archiving Session6

1. Documentation and Cataloguing in Data ArchivingSession6

2. What is documentation? Documentation: comprehensive information on the processes and methods used to produce, archive and disseminate micro-data Documentation includes metadata and other information related to the dataset

3. Role of documentation Documentation explains how the data were collected, their content and structure and any manipulation that may have taken place, how to access the data, terms for their use, etc Documentation is required in order to understand and interpret the data by providing a context: without proper documentation, data are useless The further data gets from its source, the greater the importance of the documentation (metadata) Also allows reuse of documents for future surveys

4. When to undertake documentation • Documentation is an incremental process that should be a shared responsibility among various parts of an institution • Different types of documentation can be added by different people at various stages of an information object’s life cycle • A common documentation framework, used by different actors - the actor who is closest to the information to be used as documentation/metadata adds that information to the framework

5. Types of material for documentation • Three broad categories of documentation: • Explanatory material • Contextual information • Cataloguing material

6. Types of material for documentation • Explanatory material – requiredto ensure the long-term viability and functionality of a dataset and without which full understanding of the dataset and its contents cannot be achieved • Data collection methods (data collection process including instruments used, methods employed, and how these were developed) • Structure of the dataset (information about relationships between individual files or records within the study, e.g., the number of cases and variables in each file and the number of files in the dataset) • Technical information (computer system used to generate the files; software packages with which the files were created; medium on which the data was stored; and complete list of all data files present in the dataset)

7. Types of material for documentation (contd) • Explanatory material (contd.) • Variables and values, coding and classification schemes (descriptions of all variables (or fields) in the dataset, with explanations about coding and classifications used and for blank and missing fields) • Derived variables (how it was done) • Weighting and grossing (procedures should be explained) • Data source (sources from which the data is derived e.g. questions used) • Confidentiality and anonymization (if data contain any confidential information or anonymization has been implemented and implication of both on data usage)

8. Types of material for documentation (contd) • Contextual information - the context in which the data was collected, and how it was put to use • Description of the originating project (why the data collection was felt necessary; who or what was being studied; the geographic and temporal coverage) • Provenance of the dataset (history of the data collection process, changes and developments that occurred in the data themselves and the methodology, or any adjustments made) • Serial and time-series datasets, new editions (e.g., descriptions of changes in question text, variable labelling or sampling procedures for repeated cross-section, time-series datasets)

9. Types of material for documentation (contd) • Catalogue metadata: • A sub-set of core data documentation providing standardized structured information explaining the purpose, origin, time reference, geographic location, creator, access conditions and terms of use of data

10. Metadata standards Traditionally, data producers wrote text-based codebooks. To take advantage of web technology, these have been replaced by XML-based codebooks Use of metadata standards brings key data documentation together into a single document, creating detailed and structured content about the data. This enhances: Quality of statistical documentation provided to data users Access to the data and semantic interoperability of data sets The Data Documentation Initiative (DDI) Dublin Core Metadata Standard

11. Metadata standards (contd.) On XML (eXtensive Markup Language) A way of tagging text for meaning instead of appearance (i.e., XML can be used to organize text by tagging with meaningful information Unlike text in the database, XML text files can be viewed and edited using any standard text editor With appropriate tools, XML files can be searched and queried like a regular database XML documents can be read and transformed by other software applications into user-friendly formats, e.g., spreadsheets, PDF files or web pages

12. Metadata standards (contd.) The Data Documentation Initiative (DDI): Is based around the data lifecycle model and provides specifications for a structured framework for organizing the content, presentation, transfer and preservation of metadata in the social and behavioural sciences Provides comprehensive metadata on the entire survey process and usage Facilitates point-of-origin capture of metadata Includes machine actionable elements to facilitate processing, discovery and analysis

13. Metadata standards (contd.) The Data Documentation Initiative (DDI): Facilitates reuse of common metadata items because DDI is designed around schemes (lists of items) for commonly reused information within a study, e.g., categories, code schemes, concepts, universe, etc. Items are entered once and used in multiple locations in a DDI document by referencing item in the list Reuse of items supports: Consistency and accuracy of metadata content thereby minimizing redundancy and discrepancies Internal and external implicit comparisons External registries of concepts, questions, variables, etc. Metadata driven processing

14. Metadata standards (contd.) The Data Documentation Initiative (DDI): Information in DDI schemes can be stored in external registries and used by multiple studies to support: Comparisons within and between studies Organizational consistency through use of agreed content managed in registries Designed to support easy interaction with other major standards (Dublin Core, SDMX, ISO/IEC 1179, ISO 19115) Ensures that metadata can be connected to other domains or stages of the lifecycle

15. Metadata standards (contd.) Dublin Core Metadata Standard: A general purpose metadata standard for describing digital resources related to micro-data Questionnaires Reports Manuals Data processing scripts Programs etc. Makes it easy and inexpensive to create descriptive records for information resources while providing for effective retrieval of these resources on the web or other similar networked environment

16. Metadata standards (contd.) Dublin Core Metadata Standard: Consists of 15 metadata elements:

17. What is cataloguing? Cataloguing: creation of documentation for a dataset providing standardized structured information so that searchers can easily identify and access datasets according to their needs (title of study, source, year of collection, etc)

18. Cataloguing Sharing survey micro-data with legitimate users offers many benefits, e.g., the diversity of research work, the acceptability of data, the quality of data, etc. Therefore, users should be informed about existence and characteristics of datasets Cataloguing material serves as: A bibliographic record of the dataset, allowing it to be properly acknowledged and cited in publications A formal record for long-term preservation purposes Basic instrument used for resource discovery, allowing datasets to be uniquely identified within the collection by providing appropriate information to help secondary users identify the study as useful to their purpose

19. Cataloguing (contd.) Searchable catalogues facilitate finding datasets and related metadata and increase access to datasets Use of XML-based metadata standards facilitate creation of catalogues as they are structured making them searchable Information on title of dataset, data collector(s), dates of data collection, temporal and geographic coverage, methods of data collection, sampling design and frames (if undertaken), other documentation information. Also variable names, abstracts and key words…

20. Cataloguing (contd.) Characteristics of a good survey catalogue - From the user point of view: Compliant with international metadata standard, particularly XML standards Provides detailed metadata, including at the variable level Provides user-friendly search functionalities (full text search) Provides clear information on the policy and procedure for accessing the data Provides a list and direct access to reference materials (questionnaires, manuals, reports) Includes a "search by topic" compliant with an international thesaurus

21. Cataloguing (contd.) Characteristics of a good survey catalogue - From the catalogue administrator's point of view: Provides a secure environment for storing and sharing data and metadata Provides a "users' requests" and "user's management" tool to receive and respond to data requests and information queries Provides a solution for sharing public use files and licensed files Generates admin reports on access requests received/processed; most popular surveys/documents; keywords used for searching data; etc.

22. Thank You!