200 likes | 218 Views
RDA 9th Plenary. Breakout 3, 5 April 2017 16:00-17:30. Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services”. Wrap-up and next steps (Stefano Nativi, Bridget Almas, Peter Wittenburg, Jay Pearlman, Larry Lannom, ….). Why are we here?.
E N D
RDA 9th Plenary Breakout 3, 5 April 2017 16:00-17:30 Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services” Wrap-up and next steps(Stefano Nativi, Bridget Almas, Peter Wittenburg, Jay Pearlman, Larry Lannom, ….)
Why are we here? • Data Fabric Core Components, such as the PIT and DTR, provide essential pieces of infrastructure for managing and sharing Research Data • But in order to be useful, they must be connected to provide functionality in research workflows • Brokers have traditionally served this role in SOA architectures, providing facade interfaces, mediation and abstraction
GAP Composition Abstractions and Implementations PID Data Brokers Abstract Composition Executable Workflow(s) Objet Types Processing Brokers • PID and Object typing help fill the gap • Brokering services (e.g. Data services brokering and Processing services brokering) help fill the gap.
Community Services architecture End-to-end architecture N N Client applications Client applications Mediation task Harmonization task Intermediation services N x M N + M Mediation task Mediation task Data Server M M From Human-Controlled Process to Type-Triggered Automatic Processing Community(Third-party) Data Servers
Community Services architecture Client applications Intermediation services Community services(Third-party) Data Server Client-Broker-Server architecture Web-as-a-Platform Broker(s)
Useful (architectural and Software) engineering Patterns • Separation of concerns (SoC) • separate architecture into distinct components,such that each component addresses a separate concern • Information Expert • Assign a responsibility to the component that has the information needed to fulfill it • Low coupling • Implement low dependency between the components
Brokering Services • Provide the intermediation functionalities to pass from Abstraction to Implementations and vice versa • Apply Service-Oriented Architecture approach • Utilize Web-as-a-Platform (Web 2.0 patterns) • Intermediation-as-a-Service • Third-party services: not managed by Client/Server organizations
Brokering Services • Benefits • Multi-purpose, Reusable, Participate in Composition • Sustainable through ability to evolve • Flexible, Configurable, Extensible • Challenges • Trust, Governance • New cultural and business model • Requires specialist knowledge • Types • Discovery and Access • Processing and Transformation
Future: Data and workflow integration Set 1 Set 2 Set 3 • Goal: Make the life easier for scientists who are no experts in programming and handling data • Portals: integrate data and compute workflows Data Workflow Result Preview: ✔ ✔ … Data: <PID>
Future: Data and workflow integration • Label data with PID • Label (parts of) workflows with PIDs #Load data mRNA = W1(D1) miRNA = W1(D2) #Analysis res1 = W2(mRNA, miRNA) res = W3(res1) #PID for result file = writeToFile(res) create_PID(file) #Plot for preview Plot(res) PID D1 PID D2 PID D3 PID W1 PID W2 PID W3 PID W4
Matching in federation environment Repository metadata of DO bit sequences of DO PID Resolver MD bs 1 PID record DTR 5 PID Rights DB Type Record 3 2 12 4 6 rights record 13 10 Agent 9 Y 8 7 11 Controller WF 14 Matcher 14 14 Broker Profiles Broker Broker
Scalability, efficiency, flexibility, sustainability, close to the Client needs Clients Application Community services & Interface new breed of organizations providing intermediation services to “abstract” current “implementations” and enable composition Composition Intermediation Supply system
PID Centric Data Management and Access Brokering & Mediation services
Climate data processing use case (I) Challenge: • Worsening ratio of bandwith vs. volume • Server-side processing – cultural change in scientific work Solution: • Favour support for most common processing tasks over complexity/variety • Service innovation rather than technical innovation • Understand users, train influencers, reach high quality
Climate data processing use case (II) Motivation: • multiple initiatives to build data processing solutions – some exist, some to come • variety of data sources: CMIP6, Copernicus, ... • some standardization across them, but detail differences • brokering as middleware service – not for end-users, but software agents
Climate data processing use case (III) Agent (user need) Broker component with a specific role: • Middleware service catalogs will come; defines broker role • Needed: Clear definition for broker component, respecting multiple roles • common interface for broker tasking and output format (to be fed into controller) WPS (final, preprocessing) Data sources Broker