Data Fabric and Brokering Workshop: Bridging Communities for Interoperability

1. Workshop on Brokering in Data Fabrics- communityperspectives - Garching, 2016/11/22 Participants: Wouter Addink, Stefano Nativi, Abraham Nieva De La Hidalga, Francoise Pearlman, Jay Pearlman, Tobias Wiegel, Peter Wittenburg; Remote: Bridget Almas, Donald Hobern, Larry Lannom Communities involved: Biodiversity, Natural Museums, Environmental Science, Climate Modeling, Linguistics, Cultural Heritage, Geo-Sciences Hostedby Peter Wittenburgand Kathrin Beck

2. Workshop Agenda 1. Introductions 2. Needs for data fabric and interoperability 3. Technical status of fabric and brokering 4. Key gaps to be addressed 5. Addressing the gaps 6. Program approach and system level concept 7. Demonstration cases 8. Decision on steps forward 9. Funding possibilities

3. Intro to Data Fabrics Provided by Peter Wittenburg

4. Interoperability Challenge Credit: Data Fabric IG Envri+ use Case Providedby Stefano Nativi

5. Service Brokering Pattern Traditional SOA Advanced B-SOA (N+M connections) (NxMconnections) Providedby Stefano Nativi

6. Intro to Broker Provided by Stefano Nativi

7. Elements of Fabric-Broker constructs PID - Point to and locate information (contains current state information) Types - Describes the characteristics of a class of objects – noting that objects can include services. Broker – alternative descriptions: 1. mediating data to make collections fit for a use case; 2. “brokers are structured, understood and auditable processes for working across heterogeneous data”; 3. “Actualizing the capability of resource utilization (in a broad sense)”. Collections of digital objects – aggregations that can be coupled to services for a specific purpose or goal (see next slide)

8. Use case for Museum collections (use Case 1) Providedby Donald Hobert Donald, Dimitris, Wouter and team will provide a detailed description. • Define criteria and rationale for new collection (defined by a query) that would draw from existing collections. • Building an new (virtual) collection for a particular application(s) and outcome(s). • Combine very different collections (two or more) leading to a new collection with common ontologies and geospatial distributions? The original collections are not changed. • Combine cross-collection slices into a new collection and the benefits of the outcome • Objects (virtual objects) in the collections would be looked at by the broker and mediated to enable use is a process(es) with outputs in a selected format – this would allow comparison or processing of multiple collections. • There is a controller to orchestrate and monitor the entire process. • Controller also maintains provenance, metadata, audit trails, attribution, etc through the processing of new collection. Controller addresses this by drawing on broker responding to the Controller • Demonstration: PID Service, Type registries, collection service, mediation • Impact of the data fabric construct (including broker) and capabilities. Time frame for initial use case description is a week.

9. Use Case for Coupled Model IntercomparisonProject - CMIP6 (use Case 2) Tobias and Stephan and team will provide a detailed description. • Define criteria and rationale for new collection (defined by a query) that would draw from existing collections. • Building an new (virtual) collection for a particular application(s) and outcome(s). • Combine very different collections (two or more) leading to a new collection with common ontologies and geospatial distributions? The original collections are not changed. • Combine cross-collection slices into a new collection and the benefits of the outcome • Model data and observation data (Copernicus) would be looked at by the broker and mediated to allow outputs from a processes in a selected format – which would allow comparison or processing of multiple collections. There may be multiple processes. • Process is a WPS (service) and there is a scientific payload – data and a model to give new scientific information • There is a controller to orchestrate and monitor the entire process. • Controller also maintains provenance, metadata, audit trails, attribution, etc through the processing of new collection. Controller addresses this by drawing on broker responding to the Controller • The output collections consists of the output objects plus the provenance and audit trail. • Demonstration: PID Service, Type registries, collection service, mediation • Impact of the data fabric construct (including broker) and capabilities. Time frame for initial use case description is a week.

10. Possible Activities at RDA Tobias Weigel (DKRZ) • Define a discussion at the Barcelona Plenary (session proposal) • Support a discussion sponsored by the Data Fabric and Broker Interest groups as a joint session • Blueprint/patterns for Data Fabric with brokering • Addressing also technical, organizational and business perspectives – changing the paradigm (like defining operation of clouds verses HPC) • Open discussion for domains that might be interested in participating • Working Group (WG) formation? • What are the goals – maintain a dialogue? Then do we need a working group at this point? The WG could be a focal point to look at best practices from the two use cases and broader applications beyond the first two use cases. It would be of more value if the effort matures before formation of a WG. Wait to proceed

11. Actions • Donald, et al will address use case1 (Nov 30) • Tobias, et al will address use case2 (Nov 30) • Abraham will bring topic to ENVRI+ (Nov 29) • Other communities need to spell out interests (linguistics, cultural heritage, etc.) • Stefano will address broker structure options (see slide 4) (Nov 30) • Jay and Bridget will look at joint session at RDA Barcelona Meeting (Dec 15) • Next meeting (virtual) is January 20 at 16:00 CET

12. Additional steps • Logistics • Goggle docs • Email distribution list • Meeting information • Resources for pilot demonstrations • Wouter, Peter, Dimitris