IC/DoD Content Discovery and Retrieval IPT (CDR IPT) Overview

1. IC/DoD Content Discovery and Retrieval IPT(CDR IPT) Overview Jack Lucas, DNI/CIO/I2E, Clay Robinson, OSD/NII/ES&I Co-Chairs CDR IPT 13 April 2010

2. Agenda • CDR IPT Overview • CDR IPT Architecture Model & Artifacts Overview

3. CDR IPT - Background • The Content Discovery and Retrieval Integrated Product Team • (CDR IPT) was established by joint memo signed by the • DNI Deputy CIO and the DoD CIO’s Director for • Information Policy and Integration • Committed the IC and DoD to a joint vision and • shared oversight for realizing a common • services-based environment The IC and DoD Services-based Environment

4. CDR IPT Overview • CDR IPT was established by the DNI Deputy CIO and the DoD CIO’s Director for Information Policy and Integration as a key enabler for the joint vision and shared oversight for realizing a common services-based environment • CDR IPT co-chaired by Jack Lucas, ODNI/CIO/I2E, and Clay Robinson, DoD CIO/ES&I • CDR IPT consists of representation from the IC and DoD • Purpose: • Address architectural, and engineering issues relating to content discovery and retrieval within the joint IC/DoD enterprise • Develop and publish a set of architecturally-driven standards, specifications, and applicable reference implementations to enable content discovery and retrieval • Enable greater and more flexible information sharing, primarily within and across the IC/DoD Enterprise consistent with the Federal Information Sharing Policy

5. Content Discovery and Retrieval in the Context of Mission Needs • An abstract-to-concrete model for the development of architecture elements and guidance • Each model layer provides key aspects of the overall guidance to achieve the goals and objectives of the CDR IPT • Specifications and their realizations are directly traceable to mission needs, ensuring they enable data sharing across the IC/DoD enterprise CDR Architecture Model

6. CDR IPT Developed Artifacts • CDR Requirements: defined enterprise-levelCDR requirements used to define key CDR features and components (one document) • CDR Reference Architecture: provides contextual and conceptual guidance on the Core CDR Components and their interactions with the Consumer, Provider, and Key CDR Dependency Components for architects, engineers, and developers (one document) • CDR Specification Framework: provide both CDR Service Specification developers/authors and CDR service developers/implementers guidance for realizing and implementing Core CDR Components (one document) • CDR Specifications: provide specific guidance for implementing each Core CDR Component as a technology-specific specification (six core specifications plus sub-specifications & realization guidance)

7. Development Process(Federated Search Spec) Inputs Received requirements from NCES, BCTM, LNI, DCGS-A, NGA, CIA, ICDL, others Inputs Requirements Architecture • 24 Sep 09 • 16 Oct 09 Requirements Refined Federated Search White Paper • 19 Nov 09 • 19 Nov 09 Requirements Prioritized Architecture Framework Draft CDR Reference Architecture • 19 Dec 09 • 15 May 10 Draft Framework/Specifications • Partner w/ IC/DoD • Stakeholders to Elicit • Piloting opportunities • In discussion (Pilots & Prototypes) Proof- of - Concept Update Framework/ Specifications • Post Proof-of-Concept • Est 4Qtr CY 2010 Official Recommendation • Initially Will Be an Emerging Standard IC/DoD Standards Governance & Policy (not in CDR IPT scope) • Est 1Qtr CY 2011

8. CDR Requirements • The CDR Requirements were derived from the content and discovery requirements of current programs, e.g., ICDL, LNI, DCGS-A, BCTM, DCGS-IC, NCES • 80% of the requirements were similar • Requirements were normalized • Unique requirements that were beneficial to enterprise were included in master list • Requirements were mapped to CDR core components 8

9. CDR Reference Architecture (RA) • The CDR Reference Architecture (RA) serves as the keystone CDR IPT guidance artifact and describes an overall architecture to support Enterprise-wide content discovery & retrieval • The goal of the CDR RA is to establish a flexible framework for the IC/DoD environment that is extensible, and scalable to support evolving mission/business requirements

10. Component-Driven Approach to Content Discovery and Retrieval The CDR RA identifies four main types of component groups to support Enterprise-wide content discovery and retrieval: • Core CDR Components: Search, Brokered Search, Describe, Query Management, Retrieve, Deliver • Components External to CDR Components include: • Consumer Components • Represents any entity that initiates a content discovery or content retrieval interaction with any of the Core CDR Components • Provider Components • Describes the applicable constructs related to the providers of content to support content discovery and retrieval use cases • Key CDR Dependency Components • Represents the key external dependencies needed to realize the CDR architecture

11. Core CDR Components • Search: The mechanism content providers implement to expose their content collections for discovery and accessibility • Brokered Search: The mechanism to 1) facilitate the distribution of queries to applicable/relevant content collections and, 2) aggregate the returned results • Describe (Content Collection Representation): The mechanism for content collectors to expose information to describe the context, access constraints, and current inventory status of its underlying content • Retrieve: Enables access to discovered content and is the primary mechanism for content consumers to access one or more specific content resources from a specific content collection • Deliver: Enables access to discovered content and is the primary mechanism for content consumers to reroute discovered results to a different consumer • Query Management: Enables content consumers to build queries, store them, access the results, and allow other consumers the ability to subscribe to them

12. Content Discovery in a Technology-Specific Fashion NCES 1.3 Evolution of Content Discovery Specifications & Technology IC SOA 2.4.1 NCES 3.0 CDR 1.0 • SOAP Support • Queries tightly coupled to DDMS 1.0 • Custom response • SOAP Support • Some RESTful support • Extensible query architecture • Retrieval support • Atom response • RESTful Support • Based on OpenSearch • Extensible query architecture • Retrieval support • Atom response • SOAP Support • RESTful Support • Extensible query architecture • Retrieval support • Extensible response architecture • A resilient architecture is a foundational aspect of the CDR IPT approach • Must support multiple technical approaches to meet the various needs across the IC/DoD Enterprise • A single technology does not exist that solves everyone’s needs • New technologies will be introduced that impact CDR in the future • Maintaining a strong separation of concerns is critical to maintaining future resilience • CDR does not directly cover Security, Service Discovery, Messaging, or Service Description • CDR functionality is critically dependent on these capabilities

13. Specification Framework • The CDR Specification Framework document provides guidance for ensuring consistency and interoperability in the development of CDR Service Specifications • It describes the structure and content for CDR Service Specifications including: • The description of their key properties • A decomposition of key behaviors for various environmental and technical considerations • The Specification Framework is updated incrementally to create sub-sections for each core CDR component

14. The Search Component comprises a single Search function that is responsible for three activities that underpin Content Discovery capabilities: Query Information: Search passes a recognizable, appropriately formatted query to a Search Function which triggers a set of resource metadata in response to the query. The framework does not mandate a specific query language syntax. Paging Information: Supports sequentially providing slices of resource metadata returned as results to the search consumer. Paging may respond to specific consumer instructions (e.g. using a specified starting point in a list of results). Result Presentation Information: Takes the resource metadata returned as results by search, applies appropriate formatting, and returns the formatted resource metadata to the search consumer (this is optional and may be defaulted by Specifications) Example: Search Specification Framework Essentials A Search Component’s results are resource metadata rather than actual content resources. In the context of Search, resource metadata generally refers to a subset of a resource’s available metadata, not the entire underlying record.

15. Service Specification A service specification is a formal description of a core component defined within the CDR RA • Includes an interface definition, an information model, a quality model, and a behavior model • Provides stakeholders with a description of the behavior and structure of the service capability • Provides service consumers with information to integrate with a particular service • Provides service providers the information to implement the service

16. Specification Development Keyword XQuery OGC Filter Increment I Generic SOAP-binding Query Type DDMS Imp. Profile • Search • Retrieve* REST-binding (OpenSearch) Atom Response URL SOAP * Basic approach initially Services Increment II Policy • Search – • Brokered Search • Deliver – Delivery, Dissemination, Push (via FTP, HTTP, Email, etc.) • Describe (Content Collection Representation) – describes collection • Retrieve – additional mechanisms as required • IC and DoD “Technical Policy” UCore Imp. Guide OGC Catalog Service Key CDR spec final draft complete CDR spec development in progress Planned development Increment III • Query Management – Profiles, Pub/Sub • Update existing specs 16

17. CDR Search & Retrieve Specifications Core Specification Query Types Result Sets CDR Spec. Guidance External Spec. Guidance KEY * To be added in a subsequent increment The initial set of CDR RA-compliant Specifications provides support of the two most pervasive web service approaches today: SOAP & OpenSearch CDR Specifications to Date “Technical Policy” Guidance CDR SOAP Search Specification CDR RESTful Search Specification Keyword Specification Atom Specification A9 OpenSearch Specification DDMS Implementation Guide XQuery Specification IRM.XML Implementation Guide* OpenGIS Filter Specification UCore Implementation Guide* CDR SOAP Retrieve Specification CDR RESTful Retrieve Specification

18. CDR IPT Contact Information Jack Lucas, DNI/CIO/I2E jack.n.lucas@ugov.gov Telephone: 703-874-8213 Clay Robinson, OSD/NII/ES&I clay.robinson@osd.mil Telephone: 703-602-0937 CDR IPT wiki: https://www.intelink.gov/sites/eaportal/CDRIPT/default.aspx for access contact: Mary Farley, CDR IPT Secretariat: mary.c.farley@ugov.gov

19. Back-Up

20. CDR IPT Participating Organizations • NSA • FBI • DISA/NCES • NGA/OCIO • DIA/DSS • NRO • JIEDDO • DCGS Multi-Service Execution Team (MET) Office (DMO) • US Army/BCTM • OSD/NII • DNI/CIO/ICEA • DNI/CIO/I2E