Effective discovery of geospatial data: a geospatial catalogue perspective

Washington/Northern Virginia Chapter of IEEE/GRSS Technical Meeting Wednesday, October 31, 2007 Effective discovery of geospatial data: a geospatial catalogue perspective Dr. Yuqi Bai ybai1@gmu.edu Research Assistant Professor Center for Spatial Information Science and Systems George Mason University

Contents • Geospatial data discovery problems • Geospatial data discovery systems • System architectures • Referenced Metadata standards • Referenced Catalogue Service standard • Geospatial Catalogue Federation • Case study: GMU CSISS CFS product • Main Challenges • Proposed federation strategies • Product system • Discussion • GMU CSISS CSW/CFS Applications • Summary

Background • Large volume of geospatial data • has been accumulated over the last several decades through mapping, survey and observation • Petabyte level • NASA EOSDIS project is expected to archive one petabyte per year of raw data that are distributed among data centers. • On November 20, 2003, the NASA Land Processes Distributed Active Archive Center (LP DAAC) data archive holdings crossed the one petabyte threshold in volume*. • 1 petabyte = 1*10**15 bytes = 8*10**6 Second * 1 Gb/s (~ 92.59 days) = 8*10**7 Seconds * 100 Mb/s (~925.9 days) *http://edcdaac.usgs.gov/petabyte.asp

Significant Problem and Question • Problem • Large volume of geospatial data has to be maintained in few data centers, while these data are highly needed in all research carried out by research staff, professors and students in every college, university and government agencies. End user • Question • How can users be helped in evaluating the fitness of a particular data set, among hundreds of collections and millions of granules, for user for their specific decision or assessment? geospatial data

Geospatial Data Discovery Mechanism • Overview • Step 1: • Organizing textual information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of every piece of data set • Metadata (data about data) End user discovery interface • Step 2: • Providing catalogue discovery interface for this metadata information to end users geospatial metadata • Step 3: • Enabling direct data download or customization through online software modules, or “services” geospatial data

Geospatial Metadata • The metadata required for effective data management varies with the type of data and context of use. • Standards + Profiles End user • Standards: • ISO • ISO 15836:2003 • Dublin Core metadata element set • Stage code: 60.60 (2003-11-26) • ISO 19115:2003 • Geographic information -- Metadata • Stage code: 60.60 (2003-05-08) • ISO 19115:2003/Cor 1:2006 • Stage code: 60.60 (2006-07-05) • ISO 19115-2 • Geographic information -- Metadata -- Part 2: Extensions for imagery and gridded data • Under development • Stage code: 40.00 (2007-10-25) • ISO 19119:2005 • Geographic information -- Services • Stage code: 60.60 (2005-02-10) • ISO 19139:2007 • Geographic information -- Metadata -- XML schema implementation • Stage code: 60.60 (2007-04-17) discovery interface geospatial metadata geospatial data

Geospatial Metadata (Cont.) • The metadata required for effective data management varies with the type of data and context of use. • Standards + Profiles End user • Standards: • US • FGDC-STD-001-1998 • Content Standard for Digital Geospatial Metadata • FGDC-STD-012-2002 • Content Standard for Digital Geospatial Metadata: Extensions for Remote Sensing Metadata • NASA • ECS Science Metadata discovery interface geospatial metadata geospatial data

Geospatial Metadata Discovery Interface • The discovery interface varies with the type/structure of the underlying metadata and context of use. End user • From the user’s point of view: • Simple web page navigation with no search functionality • E.g. THREDDS discovery interface geospatial metadata geospatial data

Geospatial Metadata Discovery Interface (Cont.) • The discovery interface varies with the type/structure of the underlying metadata and context of use. End user • From the user’s point of view: • Simple web page navigation with no search functionality • E.g. THREDDS • Web page navigation with limited search functionalities • E.g. NASA GCMD discovery interface geospatial metadata geospatial data

Geospatial Metadata Discovery Interface (Cont.) • The discovery interface varies with the type/structure of the underlying metadata and context of use. End user • From the user’s point of view: • Simple web page navigation with no search functionality • E.g. THREDDS • Web page navigation with limited search functionalities • E.g. NASA GCMD • Web-based GUI with enhanced search functionalities, no public API interface • E.g. EOS Data Gateway (EDG) discovery interface geospatial metadata geospatial data

Geospatial Data Discovery Process through EOS Data Gateway (EDG) System

Geospatial Metadata Discovery Interface (Cont.) • The discovery interface varies with the type/structure of the underlying metadata and context of use. End user • From the user’s point of view: • Simple web page navigation with no search functionality • E.g. THREDDS • Web page navigation with limited search functionalities • E.g. NASA GCMD • Web-based GUI with enhanced search functionalities, no public API interface • E.g. EOS Data Gateway (EDG) discovery interface geospatial metadata geospatial data LP DAAC GES DISC

Geospatial Metadata Discovery Interface (Cont.) • The discovery interface varies with the type/structure of the underlying metadata and context of use. End user • From the user’s point of view: • Simple web page navigation with no search functionality • E.g. THREDDS • Web page navigation with limited search functionalities • E.g. NASA GCMD • Web-based GUI with enhanced search functionalities, no public API interface • E.g. EOS Data Gateway (EDG) • Web-based GUI with enhanced search functionalities, with proprietary API interface • E.g. NASA ECHO • IIMSAQL Query Language GMU CSISS ECHO OGC Wrapper discovery interface ECHO Service Core geospatial metadata LP DAAC GES DISC geospatial data

Geospatial Metadata Discovery Interface (Cont.) • The discovery interface varies with the type/structure of the underlying metadata and context of use. End user • From the user’s point of view: • Simple web page navigation with no search functionality • E.g. THREDDS • Web page navigation with limited search functionalities • E.g. NASA GCMD • Web-based GUI with enhanced search functionalities, no public API interface • E.g. EOS Data Gateway (EDG) • Web-based GUI with enhanced search functionalities, with proprietary API interface • E.g. NASA ECHO • IIMSAQL Query Language • Web-based GUI with enhanced search functionalities, with open API interface • E.g. GMU CSISS/LAITS CSW Data Download GeoBrain Online Analysis System (GeOnAS) discovery interface ebRIM Wrapper OGC Core ISO Wrapper GMU CSISS OGC Catalogue Service Core geospatial metadata geospatial data 15 Terabytes Images

GMU CSISS/LAITS CSW - Designed and Developed from Aug. 2003- Support OGC CSW 2.0.1 and 2.0.2

Geospatial Catalogue Service Standard • OGC Catalogue Service is the only available standard • specifies the interfaces between clients and catalogue services through the presentation of abstract and implementation-specific models. • Catalogue Service and its clients • OGC’s perspective: • Catalogue Service supports the ability to publish and search collections of descriptive information (metadata) for data, services, and related information objects. • Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. • Catalogue services are required to support the discovery of and binding to registered information resources within an information community. End user Catalogue Service Client discovery interface Catalogue Service geospatial metadata geospatial data

Geospatial Catalogue Service Standard (Cont.) http://www.opengeospatial.org/standards/cat

GMU CSISS ECHO OGC Wrapper ebRIM OGC Core ISO ECHO Service Core GMU CSISS OGC Catalogue Core Current Status of Geospatial Catalogue System End user discovery interface geospatial metadata geospatial data

GMU CSISS ECHO OGC Wrapper ebRIM OGC Core ISO ECHO Service Core GMU CSISS OGC Catalogue Core New Problems and Questions End user discovery interface geospatial metadata geospatial data

New Problems and Questions • Different agencies have developed their own geospatial catalogues to facilitate discovery, access, and sharing of large volumes of geospatial data, either observed satellite images or simulation data. • These geospatial catalogues are becoming accessible online through their query interfaces. • Scientists who conduct multi-disciplinary research may need to search multiple catalogues in order to find the data they need. Such work is very time-consuming and tedious, especially when the catalogues may use different metadata models and catalog interface protocols. • It is very desirable if those catalogues can be integrated into a catalogue federation, which will present a well-known metadata model and interface protocol to users and hide the complexity and diversity of the affiliated catalogues behind the interface. • With the federation, users only need to work with the federated catalogue to find the data they need instead of working with catalogues individually. • Catalogue federation service - integrating multiple legacy catalogues to facilitate distributed and integrated data discovery.

GMU CSISS ECHO OGC Wrapper ebRIM OGC Core ISO ECHO Service Core GMU CSISS OGC Catalogue Core Federation Context End user discovery interface Catalogue Federation geospatial metadata geospatial data

Federation Case Study – GMU CSISS CFS System End user Discovery Interface GMU GUI Third Party System GMU CSISS Catalogue Federation Service Community Catalogues NASA ECHO GMU CSISS OGC CSW DOE Earth System Grid Simulation Data Catalogue

Federation Case Study – GMU CSISS CFS System (Cont.) • We analyzed the following aspects of each catalogue • Metadata Conceptual Model • Query Language • Communication Protocol

Federation Case Study – GMU CSISS CFS System (Cont.) • Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: • 1. Protocol Adaptation GMU CSW and the ESG catalogue support HTTP protocol (GET/POST) binding, while NASA ECHO uses SOAP to maintain the connection with the clients. The federation server should use the correct protocol when communicating with each Catalogue service. The protocol the clients may use to talk to the federation server itself is another concern. After all protocols have been defined and identified, the federation server should support protocol adaptation internally.

Federation Case Study – GMU CSISS CFS System (Cont.) • Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: • 2. Query Dispatching The federation server is responsible for dispatching a query to the affiliated catalogue services. A dispatching model should be defined to deal with the following issues: • Transparency: Whether the federation user is aware of these affiliated catalogue services and whether users can define which catalogue services are of interest in their queries. • Sequence: Whether the federation server dispatches the users’ queries to these affiliated catalogue services in a predefined sequence, whether this sequence can be changed in runtime, and whether the federation users can define this sequence in their queries.

Federation Case Study – GMU CSISS CFS System (Cont.) • Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: • 3. Query Translation: The translation of queries is another major issue. The federation has to deal with the following problems: Metadata Query Objects: The metadata objects queried using one set of query criteria may not have counterparts in another schema. For example, the federation service cannot fulfill queries for objects defined in GMU CSW and NASA ECHO for those simulation-specific metadata objects referenced only in the ESG catalogue schema. Another issue is that the same registry object has different names, in different schemes, e.g., Granule in NASA ECHO versus DataGranule in GMU CSW. Query Format: Both GMU CSW and the ESG Catalogue accept queries in OGC Filter format, while ECHO only accepts IIMSAQL format. The federation server needs to transform an individual query into the different proprietary formats. The spatial query criterion and temporal query criterion are expressed differently in the NASA ECHO granule query payload and the GMU CSW granule query payload. Query Language Functionality: Some complex query predicates in one query language cannot be identically expressed in another one. For example, the OGC Filter specification supports nested Boolean queries. Such queries can be supported at best with difficulty on ECHO IIMSAQL, and some cannot be supported at all.

Federation Case Study – GMU CSISS CFS System (Cont.) • Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues: • 4. Results Integration: Catalogue query results from multiple Catalogue Services may need to be integrated before being sent back to users. As these different sets of metadata results may not use the same schema, the rules the federation server uses to re-organize metadata information while keeping the original content should be well designed. Furthermore, whether the clients can define the format of the query result of interest and, if so, how, also needs to be addressed.

Federation Case Study – GMU CSISS CFS System (Cont.) • We propose the following federation strategies: • 1. Protocol Adaptation • As this federation is supposed to provide a single access point to multiple, autonomous information sources, it may follow the mediator-wrapper architecture, where the federation works as a mediator, and wrappers may be deployed for communicating with specific catalogue services if protocol adaptation is needed. • 2. Query Dispatching • 1) Opaque: In this scenario, the federation service fully controls the distributed query process, with the clients having no awareness of the affiliated Catalogue Services. • 2) Translucent: The federation service may expose the affiliated Catalogue Services to the users, but the users can define neither which Catalogue Services their query can be forwarded to nor the sequence of queries. • 3) Transparent: The federation service may expose the affiliated Catalogue Service to the users, and the user chooses which catalogue service shall be used for each inquiry and the order of the inquiries.

Federation Case Study – GMU CSISS CFS System (Cont.) • Proposed federation strategies: • 3. Query Translation • Query Translation in the federation has two aspects: semantic and syntactic. • A federation usually maintains a global schema that is exposed to end-users. Metadata attribute terms in user queries always follow this global schema. Before being dispatched to an underlying affiliated catalogue service, they should be transformed appropriately. This transformation logically involves four layers: metadata term, query criterion, query criteria, and query payload, as shown in the following picture.

Federation Case Study – GMU CSISS CFS System (Cont.) • Proposed Federation Strategy • 4. Query Result Integration • A federation service needs to integrate query results from multiple underlying Catalogue Services before sending them back to the clients. It may choose to implement one of three kinds of integration mechanisms. • Opaque: In this case, the federation service defines, maintains and advertises a unique information model. Each query result from affiliated Catalogue Services should, if necessary, be transformed to this information model. The original metadata information can be kept in the final transformed query results. • Translucent: The federation service does not maintain a complete, unique information model but defines a common subset of metadata objects that are supported by all the affiliated Catalogue Services, such as name, and spatial and temporal range. The federation service transforms only this part of the metadata information, while the remaining embedded original metadata information remains unchanged in the final response. • Transparent: The federation service has no role in metadata integration. All the query results from affiliated Catalogue Services are simply grouped together, keeping the original metadata formats. In this scenario, the users are supposed to analyze each result fetched from federation service, since the results may not all conform to the same schema even though grouped together in one response.

Federation Case Study – GMU CSISS CFS System (Cont.) • Federation System Architecture • The GMU CFS consists of two types of components: Mediator and Wrapper. • The Mediator is a key component of the GMU CFS. It accepts user’s queries through an OpenGIS CSW query interface. • For better modularity and sustainability, the GMU CFS has externalized the query translation and the result transformation module for ECHO to make a wrapper, i.e., OGC CSW for ECHO. This wrapper accepts client queries, in OGC Filter format, that are forwarded by the Mediator. They are transformed to ECHO IIMSAQL format by the Query Transformation module.

Federation Case Study – GMU CSISS CFS System (Cont.) • Federation System Context

Federation Case Study – GMU CSISS CFS System (Cont.) • Discussion • This GMU CFS system can integrate NASA ECHO, GMU CSW and the DOE ESG Simulation Data Catalogue. One advantage of its design is that CFS follows the OpenGIS Catalogue Service standard as the communication protocol with the underlying affiliated catalogue services. As long as new catalogue services follow this standard, they can easily be integrated into the federation system. • However, integrating new legacy catalogue services cannot be plug and play. Abstracting the specific information models, the catalogue registration mechanism, and query orchestration would be new issues to consider when scaling this federation beyond these three catalogue services.

Federation Case Study – GMU CSISS CFS System (Cont.) • Discussion (Cont.) • In the strategy described here, the specific information models need to be carefully evaluated before incorporation into the global schema and subsequent exposure to client users, when including new catalogue services. Efforts such as the ISO Technical Committee (TC) 211 Geospatial Metadata Standard 19115, FGDC Content Standard for Digital Geospatial Metadata (FGDC, 1998) and Dublin Core (DCMI, 2006) are attempting to standardize the geospatial metadata information model, but, in many cases, their use is voluntary. • A metadata crosswalk could be of help when mapping two distinct models, but is not very suitable for one-to-many mapping. • An ontology-based approach would provide a new way to create a global schema.

Federation Case Study – GMU CSISS CFS System (Cont.) • Discussion (Cont.) • New catalogue services must be registered manually. In the design presented here, the federation service discovers the underlying catalogue services at design time, rather than at run time. This strategy greatly simplifies the mechanism for federation, and lowers the complexity of implementation. • In fact, without an automatic way to integrate the information model, it does not make much sense to register the catalogue service automatically.

Publications • Journal papers • Towards a Geospatial Catalogue Federation Service • Y. Bai, L. Di, A. Chen, Y. Liu, Y. Wei. • Photogrammetric Engineering and Remote Sensing (PE&RS). 2007, Vol.73, No.6, pp.699-709. • A Taxonomy of Geospatial Services for Global Service Discovery and Interoperability • Y. Bai, L. Di. • Computers & Geosciences. (Under review). • Book chapters • Catalogue Information Models • L. Di, Y. Bai. • Encyclopedia of Geographical Information Science. • Geospatial Image Metadata Catalogue Services • Y. Bai, L. Di, A. Chen, Y. Liu, Y. Wei. • Encyclopedia_of_Geoinformatics. • Conferences • Serving Satellite Remote Sensing Data to User Community through the OGC Interoperability Protocols • L.Di, W.Yang, Y.Bai. • AGU 2005 Fall Meeting. • GEOSS Registry System: Enabling the Registering and Discovering of Geospatial Web Services Worldwide • Y. Bai, L. Di, N. Doug, Y. Wei. • AGU 2008 Fall Meeting.

GMU CSISS CSW/CFS Applications • NASA NEHEA GeoBrain project • Mobilizing NASA EOS data and information through Web service and knowledge management technologies for higher-education teaching and research • PI: Liping Di at GMU • Co-I: 9 Educational partners • Task Lead: Peisheng Zhao at GMU

GMU CSISS CSW/CFS Applications (Cont.) • NASA AIST Grid/OGC project • Integration of OGC and Grid Technologies for Earth Science Modeling and Applications • PI: Liping Di at GMU • Co-I: • Piyush Mehrotra • NASA Ames Research Center • Dean Williams • DOE LLNL • Task Lead: Aijun Chen at GMU

GMU CSISS CSW/CFS Applications (Cont.) • GEOSS Component and Service Registry • Maintenance and enhancement of the GEOSS Registry for earth observation • PI: Liping Di at GMU • Task Lead: Yuqi Bai at GMU

GMU CSISS CSW/CFS Applications (Cont.) • GEOSS GeoNetwork Clearinghouse Candidate System • Catalogue Service Interface for Web Portals • Task Lead: Yuqi Bai at GMU

Summary • Metadata and Catalogue system work behind the scenes to support geospatial data discovery • Online discovery of textual metadata information has greatly facilitated direct discovery of large volumes of geospatial data • Multiple metadata standards/profiles exist at the international level and national level • OGC specifications (baseline/profiles) are main players regarding the definition of the the Catalogue’s behavior • Heterogeneous Catalogue systems prevent further integration and interoperability • It is desirable to have a Catalogue Federation to • provide a single interface to users, while • hiding the complexity and diversity of the affiliated catalogues behind the interface • GMU CSISS Catalogue Federation product has been presented • Research findings • System design and implementation • Lessons learned • GMU CSISS CSW and CFS systems have been successfully used in several national and international projects (funded by NASA, FGDC, and GEO)

Acknowledgements • This study was supported by grants from NASA through the REASoN program (NNG04GE61A; Professor Liping Di, Principal Investigator). • I appreciate my teammates for their kind support over the last three years. • Thank you for your attention this afternoon. Happy Halloween !

Effective discovery of geospatial data: a geospatial catalogue perspective