EarthCube – Brokering A Way Forward for Global Multi-disciplinary Data Sharing

EarthCube – BrokeringA Way Forward for Global Multi-disciplinary Data Sharing Brokering Concept Award Team Steven F. Browdy http://earthcube.ning.com/groups/brokering

Agenda • Introduction of Brokering Concept Award • Short history of EarthCube brokering activities • Current brokering approaches • EuroGEOSS Broker • Brokering Principles • EarthCube Particulars • Future brokering possibilities • Future EarthCube brokering activities • Q & A

Brokering Concept Award EarthCube’s goal is “To transform the conduct of research in geosciences by supporting community-based cyberinfrastructure to integrate data and information for knowledge management across the Geosciences.” What is needed? Integration of data, tools and communities through a cyberinfrastructure Utilization of current and emerging technologies to create a transparent infrastructure for the geosciences community The EarthCube approach: “Build it and they will come” doesn’t work What gets built has to be the result of a community process, where the community is much more than the IT enthusiasts

Brokering Concept Award Our objective Develop a strategy and roadmap for evaluating promising brokering technologies in cooperation with a diverse set of US and international partners Our approach Develop and communicate a long-term strategy and roadmap for fully developing brokering as a core Earth Cube cyber-infrastructure component Demonstrate and assess several instances of brokers that mediate search and access to heterogeneous multidisciplinary data sources and services

Brokering Concept Award Expectation Engage the long tail folks, who are part of the 85% of geoscientists who will be training the next generation Bring them onboard, understanding at some level what we’re doing and what is involved, helping to change the way science is done Interact with the other EC Groups regarding the utilization of brokering Technical assessment responsibility Pilot studies with different scientific communities Hack-a-Thons in conjunction with ESIP

Brokering CA Program Plan 4/1/12 6

Short History • Roadmap development • An ongoing process • Hack-a-Thon 1 • Used EuroGEOSS Broker • Pilots • NSIDC Arctic Polar Pilot • June EarthCube Charrette • Cross-fertilization with other EarthCube groups • July EarthCube PI Meeting

Short History - Roadmap • Informed by our partners and members, as well as other EarthCube groups and the broader community of geoscientists.

Short History – Roadmap Executive (Meta)Summary • Developing infrastructure is a complex, dynamic process. (See the NSF report on Understanding Infrastructure: Dynamics, Tensions, and Design) • http://earthcube.ning.com/group/governance/forum/topics/research-review-understanding-infrastructure-dynamics-tensions • Brokers help mitigate the tensions by connecting disparate systems without imposing new burdens upon those systems

Short History – Roadmap Executive (Meta)Summary • Some of the benefits of a Brokering Framework include: • Lowers barriers to participation in distributed systems • Accelerates interconnection of disparate systems • Facilitates sustainability, reusability, extensibility, and flexibility of the infrastructure • Enhances multi-disciplinary interoperability • Removes need to impose common (e.g. federal, “top-down”) specifications enabling a more adaptive “bottom-up” evolution of the infrastructure

Short History – Roadmap Executive (Meta)Summary • Community engagement • It is necessary to very actively engage both the science and informatics communities, and to employ use-case-driven requirements • it is impossible to define all requirements in advance of development of an infrastructure • In the short term we will be addressing small community problems through collaborative pilots • limited initial capabilities targeted to specific communities • guided by use cases • ongoing user and community feedback • Evolve into larger prototypes addressing larger intercommunity problems

Short History – Roadmap Executive (Meta)Summary • Brokering has been demonstrated to be an essential part of a robust, adaptive cyberinfrastructure • Next Steps • Critical questions of governance and detailed implementation will be addressed • Continued pilots and community engagement • Expansion of brokering pilots into fully operational prototypes

Short History – Hack-a-Thon 1 • Featured the EuroGEOSS Broker • All hack-a-thon related material is on the EarthCube Brokering ning site • http://earthcube.ning.com/group/brokering/page/brokering-hack-a-thon-1 • A webinar was given to introduce the featured broker on May 10 • A period of 10 days was used to allow participants to practice with the broker • Validate services (required assistance) • Validate tools • A 2-hour WebEx-enabled Hack-a-Thon occurred, after the practice period, on May 24 • 15 participants from various EarthCube groups and science disciplines • Resulted in lessons learned and take-a-ways

Short History – Hack-a-Thon 1 • Take-a-ways • Keep the featured broker available post hack-a-thon for continued “playing” • Setup a wiki area for further discussion and collaboration • Maintain a knowledgebase for broker issues • Lessons learned • Too narrowly focused • Future hack-a-thons will include discussions from previous ones • Need a broader audience • Allow a longer practice period with telecons to maintain engagement • Poll participants for questions and comments to engage everyone

Short History - Pilots • NSIDC Arctic Polar Pilot • Using the EuroGEOSS Broker • Discovering and accessing multidisciplinary data • Analyzing the broker • Providing detailed requirements to developers at NSIDC, CNR, and NCAR to enhance capabilities

Short History – etc. • Attended June EarthCube Charrette • Much interaction with other groups • Very productive in terms of working towards a shared vision, but more work to do • Started the consensus process on priorities and timelines • Cross-fertilization with other EarthCube groups • Gave presentations about brokering to other groups • Worked together on common use cases • Understanding areas of potential success for brokering • July EarthCube PI Meeting • Consensus on reference architecture diagram • Consensus on development of common use cases

Current Brokering ApproachesSingle Monolithic Instance - Standalone

Current Brokering ApproachesSingle Monolithic Instance - Cloud

Current Brokering ApproachesMultiple Instances – Cloud Farm

Current Brokering ApproachesCommunity Deployment

Introduction to the EuroGEOSS Broker The EuroGEOSS Broker was developed by the Laboratory of Earth and Space Science Informatics (ESSI-Lab) of the Institute of Atmospheric Pollution Research (IIA) of the National Research Council of Italy (CNR) It has been deployed in many situations, such as: GEOSS Common Infrastructure NSIDC UNEP-Live WIS (WMO Information System) It’s capabilities include discovery, access, and “light” semantic mediation New “accessors” can be added to interoperate with data providers using standards currently unknown to the broker

Introduction to the EuroGEOSS BrokerSupported Sources OGC WCS 1.0, 1.1, 1.1.2 OGC WMS 1.3.0, 1.1.1 OGC WFS 1.0.0 OGC WPS 1.0.0 OGC SOS 1.0.0 OGC CSW 2.0.2 Core, AP ISO 1.0, ebRIM/CIM, ebRIM/EO, CWIC WAF Web Accessible Folders 1.0 THREDDS 1.0.1, 1.0.2 THREDDS-NCISO 1.0.1, 1.0.2 THREDDS-NCISO-PLUS 1.0.1, 1.0.2 CDI 1.04, 1.3, 1.4, 1.6 GI-cat 6.x, 7.x GBIF GeoNetwork (versions 2.2.0 and 2.4.1) catalog service HYDRO Deegree (version 2.2) catalog service • ESRI ArcGIS Geoportal (version 10) catalog service • OpenSearch 1.1 accessor • OAI-PMH 2.0 (support to ISO19139 and dublin core formats) • NetCDF-CF 1.4 • NCML-CF • NCML-OD • ISO19115-2 • GeoRSS 2.0 • GDACS • DIF • File system • SITAD (Sistema Informativo Territoriale Ambientale Diffuso) accessor • INPE

Introduction to the EuroGEOSS BrokerAvailable Catalog Interfaces OGC CSW 2.0.2 AP ISO 1.0 OGC CSW 2.0.2 ebRIM EO OGC CSW 2.0.2 ebRIM CIM ESRI GEOPORTAL 10 OAI-PMH 2.0 OpenSearch 1.1 OpenSearch 1.1 ESIP OpenSearch GENESI DR GI-cat extended interface

Broker Use Cases View the webinar for Hack-a-Thon 1 to see these use cases demonstrated: https://esipfed.webex.com/esipfed/ldr.php?AT=pb&SP=MC&rID=59212547&rKey=72865c52d24d8992

Use Case 1 – Traditional Query Example 1 Keyword Biotemperature Bounding Box North Africa Example 2 Keyword Combination AND, OR Preview via Access Broker Native Source WCS GeoTiff EPSG:4326 Accessed as WMS PNG EPSG:4326

Use Case 2 – Semantics-enabled Query Example 1 Automatic Expansion Starting term: Biodiversity Expanding Relation More Specific Example 2 User-assisted Expansion Starting from a keyword in chinese 海拔(altitude) Preview via Access Broker Native Source WFS GML EPSG:4326 Accessed as WMS PNG EPSG:4326

Use Case 3 – Access on a Common Grid Definition of a Common Grid Environment for discovered datasets Encoding Format Spatial Resolution Spatial Extension Coordinate Reference System Etc. Download datasets on the Common Grid Visualize Downloaded Datasets on the Common Grid (using the Unidata IDV tool)

Use Case 4 – Third-party Clients Discovery: GI-go (CSW/ISO Client) Firefox Search Bar (Opensearch Client) Access Mapshup (WMS Client) Preview via Access Broker Native Source WCS GeoTiff EPSG:4326 Accessed as WMS PNG (Google CRS) EPSG:3857

Brokering Framework Data Providers Resources Users Applications/portals Web 2.0Resources Web 2.0 Broker Publish Broker “Authoritative” Geospatial Resources Discovery Broker Access Broker MNL component Semantic engines Thesauri/ Gazetteers Semantic Broker

Brokering PrinciplesProblem Statement Whereas: Earth system science data are extremely diverse (in format, description, scale, precision, etc.) Different data users discover, access, interpret, and manipulate these data with a wide variety of tools and services that may be specific to the data and each user’s needs Interdisciplinary science requires people to access and use data from very different disciplines and communities Data providers are increasingly expected to serve communities outside their normal clientele

Brokering PrinciplesProblem Statement Whereas: Both users and data providers experience impediments in achieving multi-disciplinary interoperability Connectivity for multi-disciplinary interoperability should not restrict the autonomous nature of discipline-specific systems Technology evolution and maintainability is a great challenge for cyber-infrastructure sustainability and usability Brokers should be capable of improving and facilitating access to both “big data” and “long tail” resources, in real time or static environments

Brokering PrinciplesAssertions We Believe: No one information technology or standard will serve all user needs Data services (discovery, access, processing, semantics) should be openly accessible in well-defined, machine-interpretable ways Cyberinfrastructure should maximize data and service usability for both data providers and data users Flexibility is necessary for incremental expansion of data service mechanisms and to easily achieve scalability Brokers can be an effective way to achieve this flexibility

Brokering PrinciplesAssertions We Believe: Brokering [or a brokering framework] provides the cyberinfrastructure that allows providers and users to better take advantage of the open services A brokering framework is best managed and maintained by the cyberinfrastructure (rather than Data Users or Data Providers) A brokering framework should be transparent to Data Users and Data Providers Not all existing systems will continue, but it is more sustainable to supplement, not supplant,systems mandates and governance arrangements

Brokering Principles Working Definition of a Brokering Framework Definition 1: Brokers are middleware interconnecting client and server components in the EarthCube cyberinfrastructure Brokers are services facilitating the run-time interconnection (sharing of resources) among users and providers in a way that requires little effort on the part of either A brokering framework can consist of multiple brokering components to support different capabilities A brokering framework may include discovery, semantic enhancement and natural language, data access, processing and publishing

Brokering Principles Working Definition of a Brokering Framework Definition 2: A broker enables at runtime service components to transparently make and receive requests and responses in a heterogeneous and distributed environment. It is the foundation for building multi-disciplinary applications from distributed service components and for interoperability between applications in heterogeneous and homogeneous environments. Read our Brokering Principles at the Brokering Concept Award site, and feel free to post comments: http://earthcube.ning.com/group/brokering/forum/topics/brokering-principles-continued

EarthCube Particulars • Proposed reference architecture • Connection with governance • Common use cases

EarthCube ParticularsReference Architecture

EarthCube ParticularsGovernance • How should EarthCube CI brokers be deployed and maintained? • What are the requirements for deployment, who sets them, and what processes are used? • What are the requirements for maintenance, who sets them, and what processes are used? • What is the funding mechanism? • What are the policies for brokering across the boundary between communities and the EarthCube CI? • Standards for publishing data, information, and knowledge? • What is the process for the evolution and innovation of brokering for the EarthCUbe CI? • Other issues?

EarthCube ParticularsUse Cases • Implementation Goals • Interdisciplinary demonstration for diverse providers and users • “Full” range of the EarthCube CI exercised • Collaboration between EarthCube groups • Provide a clear demonstration of benefits • Can be continued as a practical implementation • Use cases being evaluated • Atmospheric ash • Gulf of Mexico hypoxia • Arctic polar environment

Atmospheric AshSolid Earth Use Case Impacts of volcanic emission on human activities (Pilot 1 – volcanic supersite) Pilot 2 – Volcanic ash from Icelandic eruption Disciplines – seismology, geodesy, polar, atmosphere, biodiversity, transportation, Potential EarthCube participants – Other

Gulf of Mexico HypoxiaWater Use Case • Water and Ocean environment • Gulf of Mexico Hypoxic region • Disciplines – land cover, river water quality, ocean chemistry, weather, biodiversity • Potential EarthCube participants – • Other?

Arctic EnvironmentPolar Use Case Arctic Environment Data exploration service for discovery of interdisciplinary Arctic data. Focus areas – arctic contaminant monitoring, seasonal sea ice predictions, biodiversity Potential EarthCube participants – Other?

Future Brokering Possibilities • Additional functionalities • Enhanced semantics (ontologies) • Multi-natural language processing • Enhanced data transformation • Interoperation with workflows • HPC deployment for modeling • Architecture/framework alternatives • Interoperability between same brokers • Interoperability between different brokers

EarthCube – Brokering A Way Forward for Global Multi-disciplinary Data Sharing