Topics

1. DDI – Metadata for social science dataWolfgang Zenk-MöltgenGESIS – Leibniz Institute for the Social SciencesWolfgang.Zenk-Moeltgen@gesis.orgDataCite Summer Meeting 2010 – Making datasets visible and accessibleHannover, 7-8 June 2010

2. Topics • About DDI • Basic DDI Concepts • Identification and Citation • Application Examples Acknowledgement: DDI Alliance TIC members, namely Wendy Thomas, Arofan Gregory, Joachim Wackerow

3. About DDI DDI – Data Documentation Initiative „The Data Documentation Initiative (DDI) is an effort to create an international standard for describing social science data. Expressed in XML, the DDI metadata specification now supports the entire life cycle of social science datasets. DDI metadata accompanies and enables data conceptualization, collection, processing, distribution, discovery, analysis, repurposing, and archiving.” (Stefan Kramer) http://www.ddialliance.org/

4. History of DDI • Concept of DDI and definition of needs grew out of the data archival community • Established in 1995 as a grant funded project, initiated and organized by ICPSR • February 2003 – Formation of DDI Alliance • Membership based alliance • Formalized development procedures

5. Members of DDI University of Alberta, Canada Australian Bureau of Statistics (ABS) Australian Social Science Data Archive (ASSDA) University of California, Berkeley -- Computer-Assisted Survey Methods Program and UCDATA University of California, California Digital Library Centro De Investigaciones Sociologicas (CIS), Spain CEPS/INSTEAD -- Luxembourg Cornell University (CISER) Danish Data Archive Data Archiving and Networked Services (DANS), The Netherlands Finnish Social Science Data Archive German Socio-Economic Panel Study (SOEP) GESIS - Leibniz Institute for the Social Sciences University of Guelph Institute for Quantitative Social Science (IQSS) at Harvard University Institute for the Study of Labor (IZA) Inter-university Consortium for Political and Social Research (ICPSR) Massachusetts Institute of Technology (MIT) University of Minnesota, Minnesota Population Center National Opinion Research Center (NORC) Norwegian Social Science Data Service (NSD) Open Data Foundation Princeton University Research Data Centre of the German Federal Employment Agency, Institute for Employment Research (IAB) Roper Center Stanford University Survey Research Operations, University of Michigan Swedish National Data Service (SND) Swiss Foundation for Research in Social Sciences (FORS) United Kingdom Data Archive University of Toronto University of Wisconsin U.S. Bureau of Labor Statistics (Associate Member) World Bank, Development Data Group (DECDG) Yale University • Initial members • Social science data archives • Statistical data producers • Actual membership expanded by • Research data centers • Data producers • Commercial organizations

6. DDI is being used around the world • Archives and Data Libraries • Research Institutes and Data Service Centers • International Organizations and National Statistical Agencies

7. DDI Versions • 2000 – DDI 1.0 • Documentation of simple surveys, microdata only • 2003 – DDI 2.0 and 2.1 • Extension to aggregate data • Support for geographic material • 2008 – DDI 3.0 • Lifecycle model: Shift from the codebook centric / variable centric model to capturing the lifecycle of data • Focus on metadata creation and re-use • “Machine-actionable” aspects of DDI to support programming • CAI instruments supported by expanded description of the questionnaire • Data series support (longitudinal surveys, panel studies, etc.) • Support comparison by design and comparison-after-the-fact • Improved support for describing complex data files • 2009 – DDI 3.1 • Correction of bugs • Introduction of final URN structure to ensure persistent URNs for all identified elements

8. Basic DDI 3 Concepts • Lifecycle Concept • Re-usable documentation • Modules • Maintainables, versionables, identifiables • Scheme-based (maintainable lists) • Relations to other standards • Controlled Vocabularies

9. The Data Life Cycle Archiving Concept Collection Processing Distribution Discovery Analysis Repurposing

10. DDI 3 versus earlier versions Previous versions had the “codebook” idea that creates a documentation of a social science dataset DDI 3 with it’s lifecycle model allows for documentation at all stages from study conception and data processing until analysis and repurposing of data DDI 3 uses XML Schemas instead of XML Data Type Definition (DTD) to have a stronger definition of metadata types, to make better reuse of content and to reach the goal of “machine actionability” A DDI 3 instance includes now the “simple instance” from previous DDI versions. Multiple data products can be included for a single study.

11. DDI 3.1 Modules Contain groups of related documentation elements Some are related to the Lifecycle model, some are technically grouped • Archive module • Comparative module • Conceptual components module • Data collection module • Dataset module • Dublin Core Elements module • DDI profile module • Grouping module • Instance module • Logical product module • Physical data product module • (plus inline n-cube, normal n-cube, tabular n-cube module and proprietary module) • Physical instance module • Reusable module • Study unit module

12. Usage of DDI 3 Modules Study Unit • Identification • Coverage • Topical • Temporal • Spatial • Conceptual Components • Universe • Concept • Representation (optional replication) • Purpose, Abstract, Proposal, Funding Data Collection • Methodology • Question Scheme • Question • Response domain • Instrument • using Control Construct Scheme • Coding Instructions • question to raw data • raw data to public file • Interviewer Instructions Logical Product • Category Schemes • Coding Schemes • Variables • NCubes • Variable and NCube Groups • Data Relationships Physical Data Structure • Links to Data Relationships • Links to Variable or NCube Coordinate • Description of physical storage structure • in-line, fixed, delimited or proprietary Physical Instance • One-to-one relationship with a data file • Coverage constraints • Variable and category statistics Archive • Organization or individual which has control over the metadata • Lifecycle events • Archive specific information etc…

13. Maintainables, Versionables, Identifiables Inheritance Maintainables (may be maintained separately, need agency) Versionables (may be versioned in the form „1.0.0“) Identifiables (may be identified and be referenced, either by ID or URN) Inheritance Other DDI elements

14. DDI Schemes Schemes = Lists of elements of one type Examples archive OrganizationScheme datacollection QuestionScheme ControlConstructScheme InterviewerInstructionScheme conceptualcomponent ConceptScheme UniverseScheme GeographicStructureScheme GeographicLocationScheme logicalproduct CategoryScheme CodeScheme VariableScheme NCubeScheme physicaldataproduct PhysicalStructureScheme RecordLayoutScheme

15. Relationship to Other Standards Dublin Core Basic bibliographic citation information Basic holdings and format information METS Upper level descriptive information for managing digital objects Provides specified structures for domain specific metadata OAIS Reference model for the archival lifecycle PREMIS Supports and documents the digital preservation process ISO 19115 – Geography (FGDC) Metadata structure for describing geographic feature files such as shape, boundary, or map image files and their associated attributes ISO/IEC 11179 International standard for representing metadata in a Metadata Registry Consists of a hierarchy of “concepts” with associated properties for each concept SDMX Exchange of statistical information (time series/indicators) Supports metadata capture as well as implementation of registries

16. Contr. Vocab • Not part of standard • Recommendations on: • Example: TimeMethod may be • Longitudinal (Cohort or Trend) • Panel (Continuous or Interval) • TimeSeries (Continuous or Discrete) • CrossSectional • CrossSectionalAdHocFollowUp • Other

17. Identification in DDI 3 • Two possibilities to identify an element: • Specify the <ID> Tag • Agency and Version are inherited • Use the specially-structured URN • Agency and Version must be included • The structured URN approach is preferred • These IDs/URNs can be referenced • Both ways need a resolver service that turns the names into locations to make effective re-use possible • DDI Alliance ist currently working on that, based on the DNS (Domain Name System) infrastructure approach

18. URN Identification Examples URN of a maintained object To identify of a variable scheme in DDI 3 via a URN would be as follows: urn=“urn:ddi:us.icpsr:VariableScheme.V_GENDER_SCHEME.1.0.0” URN of an versionable object All versionable objects are contained within maintainable objects. To identify a variable in DDI 3 via a URN would be as follows: urn=“urn:ddi:us.icpsr.VariableScheme. V_GENDER_SCHEME.1.0.0:Variable.Gender.1.0.0” URN of an identifiable object An identifiable object may be a direct child of a maintainable object or be contained by a versionable object within a maintainable object. The full path should be provided to facilitate locating the item when referenced. <DataCollection isMaintainable=”true” id=”DC_5698” version=”2.4.0”> <Methodology isVersionable=”true” id=”Meth_Type_1” version=”1.0.0”> <TimeMethod isIdentifiable=”true” id=”TM_1”> To identify the identifiable object in the above hierarchy in DDI 3 via a URN would be as follows: urn=“urn:ddi:us.icpsr:DataCollection.DC_5698.2.4.0:TimeMethod.TM_1.1.0.0” (from the DDI Technical Specification Part I)

19. Citation in DDI

20. OtherMaterial • Elements • Citation holds full citation information for the external object • ExternalURLReference location of the external object • ExternalURNReference URN expression for the external object • MIMEType the standard internet MIME type for applications • Relationship reference to DDI object and description of relation to it • Segment specifies part of external object (e.g. with audio/video files) • UserID unique ID of other types, e.g. DOI • Attributes • Action used for local overrides in case of inheritance ("Add" | "Update" | "Delete") • id DDI ID of the element • isIdentifiable fixed value of "true" • objectSource source name or location • type required type code for type of the external object • urn DDI URN of the element • xml:lang optional identification of the language of the external object

21. DOIs and DDI URNs • Relationship still unclear • DDI URN resolution service still needed • Every identifiable element could be registered with a DOI, that would result in huge amounts of DOIs • Only study level could be registered with a DOI, e.g. each StudyUnit • In DDI all registered DOIs should be documented • Vice versa each DOI should contain the DDI URN in the metadata • Diverse software applications will make use of them

22. Application Examples • Enhanced Publications • Providing Information to connect Publications with the underlying datasets/variables used • Making retrieval of research with specific datasets/variables possible • Version History of Datasets • Documenting errata and correction history • Making it easy to cite used data

23. Supporting Enhanced Publications DDI Alliance Publications with References to Data: DDI 3.1 URN contains: Agency Object Version find agency gesis.de.ddi return resolver address http://resolve.gesis.org Publication with References (URNs) find object return URL http://www.gesis.org/doc/docxyz request document URL of Documentation and/or Data return document urn:ddi:de.gesis:VariableScheme.ZA3811_VarSch.1.0.0:Variable.V8.1.0.0

24. Supporting Enhanced Publications DSDM DDI 3 EPE Simple Export Wizard 1.2.0

25. Enhancing Publications - DatapluS A University of Tilburg and Centerdata project, supported by GESIS and the European Values Study

26. Version History of Datasets • The GESIS data catalogue holds study descriptions with links to data access • GESIS currently introduces a common versioning policy for datasets • Starting with version 1.0.0 and increasing the major, minor or revision number according to change in the dataset • Corresponding to each published version a DOI will be created • That gives transparancy in the history of data processing • Citation of used datasets will include the specific version to ease replication

27. Data Catalogue

28. Thank you!