1 / 101

A Workflow Bus for e-Science Applications

A Workflow Bus for e-Science Applications. Dr Zhiming Zhao Faculty of Science, University of Amsterdam VL-e SP 2.5. Outline. Introduction Scientific workflow in e-Science Scientific workflow management systems A workflow bus and generic e-Science framework Basic idea Related work

karim
Download Presentation

A Workflow Bus for e-Science Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Workflow Bus for e-Science Applications Dr Zhiming Zhao Faculty of Science, University of Amsterdam VL-e SP 2.5

  2. Outline • Introduction • Scientific workflow in e-Science • Scientific workflow management systems • A workflow bus and generic e-Science framework • Basic idea • Related work • Our approach • Prototype • State • Discussion • Conclusions • Future work

  3. Step2: performing the experiment Step3: analyzing the experiment results Step1: designing an experiment success Scientific experiments and e-Science • Required services: • Managing data, computing tasks, information, and meta data • Managing experiment routines (scientific workflow)

  4. Scientific Workflows in e-Science Experiment processes Temporal workflows for administration, e.g., AAA, and other issues. Workflows for: • Different levels • Different function • Different goals Abstract workflows Executable (concrete workflows)

  5. Scientific workflow management systems (SWMS) • A SWMS is able to: • Automate experiment routines • Rapid prototype experimental computing systems • Hide integration details between resources • Manage experiment lifecycle • … • A number of SWMSs have been developed: • Managing direct computing/data tasks: CondorG • Managing experiment tasks at an abstract level: Pegasus • Extending a Problem Solving Environment: Kepler • Coupling different domains software resources: Taverna

  6. Insight a Scientific Workflow Management System In our view, a SWMS at least implements: • A model for describing workflows; • An engine for executing/managing workflows; • Different levels of support for a user to compose, execute and control a workflow. Workflow (based on certain model) Composition A SWMS User support Engine level control Engine Resource level control resources

  7. Diversity in SWMS • Originated from different domains: high energy physics, bio-informatics • Different workflow model and description language: DAG, MoML… • Different execution models (engines): based on Data flow, Continuous time, process net. • Different types of user support: composition, dynamic support • Different interface for coupling resources: web services, Ptolemy middleware, and CORBA

  8. Diversity in SWMS • Taverna: • Web services based language: Scufl; • FreeFluo: engine • Graphical viz of workflow Triana Kepler

  9. Why generic framework? A generic workflow system: a SWMS can support applications from different domains. • Improve the reuse of workflow components and the workflows for different experiments • Reduce the learning cost for different systems • Allow application users to work on a consistent environment when underlying infrastructure changed

  10. Application Layer Virtual Lab (VLAM-G). Grid Layer Research context and missions Generic framework for different domains • Different levels of abstraction • Workflow services: • Short term • Long term

  11. Possible options • Abstract approach • Extend approach • Aggregate approach S1 S2 S3 SG S1 S2 S3 SG S1 S2 S3 S1 S2 S3 SG

  12. Why we choose an aggregation approach? • Abstract approach • Build a perfect system • Difficult to find a set of systems cover all the required generic functionality; it requires re-implementation of existing things • Extend approach • Incrementally development • The solution depends on that system • Aggregate approach • Maximize the reuse of the existing workflow systems • Has to handle interoperability issues; provide customized interface existing workflow system

  13. A workflow bus paradigm Workflow Sub workflow 3 Sub workflow 1 Sub workflow 2 Triana Kepler Taverna Workflow bus

  14. What is a workflow bus? It is a special workflow engine for executing meta workflows, in which sub workflows will be executed by different other engines. It allows a scientist to: • design a workflow using resources from different workflow systems • work on a SWMS which he is familiar with, but to exchange information with the other SWMSs transparently

  15. Related work Interoperability among workflow systems (sister Link project) • Resource level: Kepler invokes Taverna’s resources

  16. An agent based prototype • Why choosing agent technologies • Decompose and encapsulate control intelligence • Ontology enabled communication • The basic idea • Handle workflow/sub-workflow using different levels agents • A study manager • A number of scenario mangers • Use FIPA compliant agent implementation: JADE • Use a Ptolemy as the basic framework for supporting workflow execution

  17. Taverna Engine Triana Engine Scenario Mnger Scenario Mnger Scenario Mnger Study Mnger JADE agent framework Architecture Schedule sub-workflows • The execution of a workflow is one study, and the execution of a subworkflow is called a sub-study, or a scenario • At one level, only one study manager active • Study manager: an agent for scheduling sub workflows, and coordinating their executions via Scenario Managers • Scenario managers: agents for interfacing third party workflow engines and for reacting to Study manager

  18. Taverna Engine Triana Engine JADE agent framework User front end Scenario Mnger Scenario Mnger Scenario Mnger Study Mnger Ptolemy Actor Actor Actor Director

  19. How it works • In user front end: a user defines meta workflow, each actor represents a sub workflow • At runtime, each actor initiates a scenario agent, and passes the workflow description to the scenario manager • A scenario manager controls an engine and execute the sub-workflow

  20. Prototype • Ptolemy II 5.0.1 • JADE • Tested with Taverna, Triana engines

  21. Experiment results • Communication cost

  22. Cont. • Scalability

  23. Cont. • Overhead

  24. Applications • Use case 1: • A user has workflow in Taverna • Some functionality is missing in Taverna but can be provided by Triana • He can develop the workflow in two systems, and run it via the workflow bus • Use case 2: • A user wants to execute a Taverna or Triana workflow in multiple instances with different input data

  25. States • Taverna, Triana have been tested • MoML is used for describing meta workflows

  26. Discussion • Challenges in scientific workflow support • Domain specific application: specific requirements on experiment scenarios • Generic workflow support and domain specific applications • Existing workflow management systems are diverse in functionality, design and user support

  27. Conclusions • A workflow bus is practically feasible approach to realize generic e-Science framework • Multi agent technology provide a distributed environment for decompose and encapsulate control intelligence • Ptolemy II provides different computing paradigms which give user freedom to execute workflows

  28. Future work • Working on developing a scenario manager for Kepler engine • Synchronized data flow is currently used. More computing modes will be evaluated. • Provenance for workflow bus

  29. References • Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology (CIT2005), pp. 339-345 . IEEE Computer Society Press, Shanghai, China, September 2005. • Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows in conjunction with the 5th International Conference on Quality Software, pp. 357-364. IEEE Computer Society Press, Melbourne, Australia , September 19th-21st 2005. • Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent Technology and Generic Workflow Management in an e-Science Environment, in Hai Zhuge and G.C. Fox, editors, Grid and Cooperative Computing - GCC 2005: 4th International Conference, Beijing, China, in series Lecture Notes in Computer Science, vol. 3795, pp. 480-485. Springer, November 2005. ISBN 3-540-30510-6. (DOI: 10.1007/11590354_61) • Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence (ICTAI05), pp. 19-23. IEEE Computer Society Press, Hongkong, China, November 14th-16th 2005.

  30. A workflow bus paradigm A scientific workflow Sub workflow 2 Sub workflow 3 Sub workflow 1 SWMS 1 SWMS 2 SWMS 3 Workflow bus

  31. Architecture Workflow bus prototype Study manager Scenario manager Taverna Triana

  32. BPEL: an introductory(some slides are from Oracle) Z. Zhao

  33. A big picture Using IT technologies (software and hardware) to enhance (automate) business processes!

  34. Business process • What is a business process: • From business point view: interaction scenarios among partners; • From WS point of view: contracts among service providers; • Business processes can be described in two ways. • Executable business processes model actual behavior of a participant in a business interaction. • Business protocols, in contrast, use process descriptions that specify the mutually visible message exchange behavior of each of the parties involved in the protocol, without revealing their internal behavior. • Implementing business processes using Web services!

  35. Web Services Meet Business Processes Web Service 1 Web Service 4 Web Service 2 Web Service 5 Web Service 3 Web Service n

  36. CreditResponse InventoryResponse CreditCheck Purchase Order ReserveInventory Invoice Example Problem Space Credit Service Consolidate Results PO Service Client Inventory Service

  37. Sample Business Process: Purchase Order Sample Purchase Order Purchase Order Request Business“A” Business “B” Purchase Order Acknowledgement Purchase Order Response

  38. Coordinate asynchronous communication between services Correlate message exchanges between parties Implement parallel processing of activities . . . Manipulate/transform data between partner interactions Support for long running business transactions and activities Provide consistent exception handling . . . Business Process Challenges

  39. Orchestration vs Choreography • Orchestration • An executable business process describing a flow from the perspective and under control of a single endpoint (commonly: Workflow) • Choreography • The observable public exchange of messages, rules of interaction and agreements between two or more business process endpoints

  40. Send PO Receive PO Receive PO Ack Send PO Ack Receive PO Response Send PO Response From a Choreography Perspective Public Process Business A Business B PO Request PO Acknowledgement PO Response Choreography – The observable public exchange of messages

  41. From an Orchestration Perspective Private Process Business A BPEL Workflow Transform Send PO PO Request From ERP Receive PO Ack PO Acknowledgement To ERP Transform Receive PO Response PO Response Orchestration – A private executable business process

  42. Business Analyst Tool BusinessA BusinessB Orchestration and Choreography Together Generate BPEL Template Generate BPEL Template Business B BPEL Workflow Business A BPEL Workflow Receive PO Transform Transform Send PO PO Request Send PO Ack Receive PO Ack PO Acknowledgement Receive PO Response Transform Transform Receive PO Response PO Response Two BPEL workflow templates reflecting a business agreement

  43. Recent History of Business Process Standards BPSS (Business Process Specification Schema) (ebXML) BPML (Business process modeling language) (Intallio et al) WSCI (Web service choreography interface) (Sun et al) WS-Choreography (W3C) 2000/05 2001/03 2001/05 2001/06 2002/03 2002/06 2002/08 2003/01 2003/04 XLang (BizTalk) (Microsoft) WSFL (Web Service Flow Language) (IBM) WSCL (Web service Conversation language) (HP) BPEL4WS 1.0 (Business process Execution language 4 Web service)(IBM, Microsoft) BPEL4WS 1.1(OASIS)

  44. Standards • Business Process Execution Language (BPEL). • BPEL 1.1 2003 • BPEL is promoted as a standard WSBPEL through Organization for the Advancement of Structured Information Standard (OASIS); • BPEL was developed through the combination of IBM's WSFL and Microsoft's XLANG. • Also known as BPEL4WS (the original proposal) and WSBPEL (the OASIS standard being developed). • Business Process Specification Schema (BPSS) from ebXML. • Business Process Management Initiative BPMI from OMG. • Web Services Choreography Description Language (WS-CDL) from W3C focuses on describing peer-to-peer collaborations between Web service participants by defining their behaviour from a global viewpoint. • XML Process Definition Language (XPDL) from WfMC • Process Specification Language (PSL). • …

  45. BPEL4WS • BPEL4WS provides a language for the formal specification of business processes and business interaction protocols. • It defines a new web service by composing a set of existing services: • Describe a business process • Implement a process • It is positioned to become the Web services standard for composition.

  46. Business Process Execution Language for Web Services • Version 1.0 released by IBM, Microsoft and BEA in August 2002 • Accompanied by WS-Coordination, WS-Transaction which remain unsubmitted to standards bodies • Version 1.1 submitted to OASIS April 2003 • XML language for describing business processes based on Web services • Convergence of XLANG (Microsoft) and WSFL (IBM) • Unprecedented industry consensus • IBM, Microsoft, Oracle, Sun, BEA, SAP, Siebel …

  47. Choreography - CDL4WS Orchestration - BPEL4WS Standards Building Blocks of BPEL BusinessProcesses Transactions Quality ofService Coordination WS-Security WS-Reliability Context Management Discovery UDDI Description Description WSDL SOAP Message XML Transport HTTP,IIOP, JMS, SMTP

  48. BPEL Depends on WSDL and WSDL Extensions Service Implementation Definition Service Port Service Interface Definition Binding Port types define Operations Message Type

  49. BPEL Scenario Structure <process> <!– Definition and roles of process participants --> <partnerLinks> ... </partnerLinks> <!- Data/state used within the process --> <variables> ... </variables> <!- Properties that enable conversations --> <correlationSets> ... </correlationSets> <!- Exception handling --> <faultHandlers> ... </faultHandlers> <!- Error recovery – undoing actions --> <compensationHandlers> ... </compensationHandlers> <!- Concurrent events with process itself --> <eventHandlers> ... </eventHandlers> <!- Business process flow --> (activities)* </process>

  50. Primitive Activities <invoke> <receive> <assign> <reply> <throw> <terminate> <wait> Structured Activities <sequence> <switch> <pick> <flow> <link> <while> <scope> BPEL Activities

More Related