Realize an e-Science workflow bus: technical issues and agenda

1. Realize an e-Science workflow bus: technical issues and agenda Zhiming Zhao VL-e SP2.5 weekly meeting

2. Outline • Some recent events • Workflow bus • Concept and challenges • Typical scenario • An agent based design • Agenda

3. Recent events: Krakow Grid workshop • Background • Since 2001 • Organized by the Polish Grid community: AGH, Cyfronet, etc., • Invited talks, oral and posters • Different tracks: security, performance, applications, • Our work • Z. Zhao, D. Vasunin, A. Wibisono et al., “Privacy issues in integrating R environment in scientific workflows”: oral presentation • In the Grid security track, and paper submission 1/Dec. • Contacts • Met some Grid security experts, e.g., Syed Naqvi (CCLRC Rutherford Appleton Laboratory, UK), in the PC of WSES 07. • Interesting talk • From EU commission • From different projects: EGEE, KF-Grid, D-Grid,

4. Cont. the 20th CODATA Int’l Conf. • Background • CODATA: Int’l org on Data for Science and Technology • 20th meeting, in Beijing • Invited talk, abstracts and posters • Keynote talk: Toney Hey • Our talk • On behalf of Bob: “Scientific Workflow management in Virtual Laboratory for e-Science (VL-e)” • Workflow support in VL-e • Contacts • Prof. Yan Baoping, director of e-Science project in Chinese academy of sciences • Interested in research cooperation • Prof. Huang Lican, P2P service discovery

5. Scientific experiment: a meta workflow Sub workflow 1 Sub workflow 2 Sub workflow 3 Sub workflow 4 Triana Kepler VLAMG Taverna Workflow bus: provide services for 1) Interoperability and integration, 2) composition, 3) provenance, 4) Enactment, 5) Human in the loop computing Generic Grid middleware An e-Science workflow bus

6. Research lines • Integration and interoperability • Composition support for meta workflows • Experiment provenance via a workflow bus • Workflow enactment and different computing models • Interactive workflow execution and human in the loop computing

7. Research lines • Prototype a basic workflow bus: Integration and interoperability • Composition support for meta workflow • Experiment provenance via workflow bus • Workflow enactment and different computing models • Interactive workflow execution and human in the loop computing

8. The basic idea • Decompose intelligence, e.g., integration, coordination, monitoring and runtime control, into scenario managers, study manager and user interfaces. • Distributed components are loosely coupled via a runtime infrastructure • Z. Zhao et. al., VLWF-BUS: a workflow bus for multi e-Science domains, IEEE Int’l Conf. e-Science and Grid computing. Meta Workflow WF 1 WF 2 Scenario manager Scenario manager Study manager Control interface Composition interface Other e-Sci. services Workflow runtime infrastructure

9. A basic set of components: for the first prototype • Workflow runtime infrastructure (WF-RTI): • Coupling distributed components • Handling data distribution: messages, data objects and files • Interfacing other e-Science services: semantic tools, Data base, etc. • Scenario manager: • Wrapping legacy workflow engines • Coordinating the execution of the sub-workflow • Study manager • Executing meta workflows • Orchestrating scenario managers, and providing runtime control • Control interface • Monitoring workflow execution • Controlling runtime behavior • Composition interface • Composing meta workflow • Providing tools for (semi) automatic composition • Validating the meta workflow

10. A typical runtime scenario Composition Control Study Scenario Scenario Meta WF Meta WF Meta WF WF1 WF1 Data location WF2 WF2 Data location Data location

11. Cont. • Issues: • How does a study manager find a proper workflow engine and assign a scenario manager? • How does a study manager manage the application boundary (talk to correct scenario manager in a correct workflow)? • How does a scenario manager decide where to store results of the sub workflow? • How will a study manager be fault tolerant: when sub workflow failed, or in possible deadlock situations? • How will scenario managers and study manager handle privacy issues and X display? Composition Control Study Scenario Scenario Meta WF Meta WF WF1 Data location WF2 Data location Data location

12. Design considerations • Middleware for message passing and data distribution • CORBA/HLA, MPI/SOAP or FIPA • Standardized interface • The integration paradigm between components • C/S, Federated • Flexible • Control intelligence realization • Customized implementation or use AI based framework

13. Agent based design • Scenario manager, study manager and user interface are encapsulated as agents • Functionality of legacy workflow engine and behavior of scenario manager are explicitly described • Study manager implements intelligences for scheduling and orchestrating scenario managers • Study manager is able to talk to generic Grid services and to enact (execution planning) workflows • A multi agent framework is used as workflow runtime infrastructure (WF-RTI) • Runtime information of the agents and workflows are distributed via the agent framework • The agent framework provides feasible infrastructure for realizing provenance and monitoring functionality

14. What is an agent? Sensors Brain Effectors • Agent is an intelligent component: • Behavior: • sensors • Effectors • Internal activities • Brain: • World model • Identity • Task description • Capability • Constraints of activity • Intelligence • Activity scheduling mechanism • Interaction strategies World model Capability Control intelligence Internal activities

15. Design paradigm Identification Agent Brain World model Task description Work flow Sensor Knowledge base Observations Effector Capability Gui Description Scenario manager Study manager Kepler SceMnger Taverna SceMnger Triana SceMnger

16. Scenario manager: sub-workflow engine wrapper Wraps sub-workflow engine and provide uniform interface for Study manager • World model: • Profile: ID (unique), name, • workflow description (content, or handle), • Brain (World model of the entire system, execution state of the sub-workflow) • Capability • Control operations for workflow engine • wf_read/wf_clearn, wf_start/wf_pause/wf_stop/wf_restart, wf_save/wf_restore, data_input/data_output • Engine control operation • eg_load/eg_kill/eg_migrate • Scenario manager/ study manager: • scen_start, scen_stop, scen_migrate • Scenario manager/ Scenario manager: • query state, inform data location, • Intelligence • Be able to start engine, and migrate it at runtime • Be able to detect the runtime state of the other scenarios and pass data

17. Study manager: Workflow coordinator Coordinates the execution of sub-workflows • Word model • Profile (ID, Type, Name, Location) • Workflow (description) • Brain: • Execution plan (schedule plan, execution model) • Knowledge backbone • World model • Capability • Study manager/ composition: execute_workflow • Study manager/ control interface: query_state, • Study manager/scenario manger: • workflow execution, • scenario manager control, • Sub-workflow execution, • Intelligence • Execute a workflow • Coordinate sub-workflows

18. Decoupled interface • Composition agent: compose workflow and activate the study manager • Runtime control agent: check the runtime information

19. Implementation agenda • What do we have now • Suresh’s work: integration study • Conceptual study on workflow interoperability • Version 1: basic prototype • Taverna and Kepler will be chosen as test case • Study manager, scenario manager and user interface will be integrated • April/2007: demonstrate the prototype, performance study, technique report (or publication). • Based on Version 1, the system will be improved incrementally • Version 2: implement application use case • Ketan’s work • IBU Marco Roos’s case • Version 3: with composition support • Results from KF-Grid • Version 4: data provenance • Cooperation with Ikay, Victor • Version 5: flexible execution