Orchestrating Scientific Workflows with BPEL

1. UCL Department of Computer Science Orchestrating Scientific Workflows with BPEL Open Middleware Infrastructure Institute University College London Software Systems Engineering Group 15 September - 16 September, e-Science Institute Edinburgh

2. UCL Department of Computer Science Contacts

3. UCL Department of Computer Science Introduction • BPEL is the industry standard for orchestration of Web services • BPEL can provide many benefits for scientific workflows • BPEL relies on complex set of technologies • Additional support required to obtain benefits • Additional language abstractions • Tool support • OMII-BPEL project offers needed additional language abstractions and tool support to hide some of the complexity • Aim is to give introduction to BPEL and OMII-BPEL in order to enable audience to express their workflow needs in BPEL using our environment

4. UCL Department of Computer Science Learning Objectives • Understand requirements of scientific workflows • Learn about basic concepts and language constructs of BPEL • Learn about advanced BPEL concepts required for large-scale workflows • E.g., hierarchical composition, asynchronous invocation, message correlation, etc. • Learn about language extensions provided by OMII-BPEL and how to use them • Learn about issues of scalability and how to deal with them • Some tips & tricks for building orchestratable Web services

5. UCL Department of Computer Science Assumptions • You understand XML and its related technologies • XML Schema 1.0 • XML Namespaces • XPath 1.0 • SOAP 1.2 • You know about Web services • WSDL 1.1

6. UCL Department of Computer Science Agenda • Overview of OMII-BPEL Offering • Introduction to workflows • Fundamental BPEL concepts • BPEL Language Elements • Advanced BPEL Concepts • Sedna BPEL Extensions • Scalability Considerations • Engineering Orchestratable Web Services • Summary, Conclusions, Discussion, Q&A

7. UCL Department of Computer Science Overview of OMII-BPEL Offering • Sedna • ActiveBPEL Engine • ActiveBPEL Management & Monitoring Console

8. UCL Department of Computer Science 1. Overview of OMII-BPEL Offering • Aim is to make benefits of BPEL accessible to application scientists • For this, need to overcome a number of issues: • Provide suitable set of abstractions to simplify creation of scientific workflows • Provide tool support to hide complexity of technologies inherent to BPEL • Provide integration of various middleware technologies • This is what OMII-BPEL offers

9. UCL Department of Computer Science Overview of OMII-BPEL Offering (continued)

10. UCL Department of Computer Science Sedna • Our visual modelling environment • Provides access to all native BPEL activities plus Sedna BPEL extensions • Developed as Eclipse plug-in • Features include: • Pre-deployment validation • Integration with ActiveBPEL engine • Project management including cvs • Context-sensitive wizards

11. UCL Department of Computer Science Sedna (continued) • Package Explorer for managing workflow projects and their related resources • Integration with Eclipse New Wizard • Simple Project • New BPEL Workflow Wizard • Sedna files

12. UCL Department of Computer Science Sedna (continued) • Overwiew Page • Wizards for setting up partners, global variables, and namespaces • Wizards to inspect and modify setup information

13. UCL Department of Computer Science Sedna (continued) • Process Map • Palette of activities • Canvas • Properties view • Problems view • Deployment menu option • BPR archive

14. UCL Department of Computer Science ActiveBPEL Engine • Open source implementation of BPEL4WS 1.1 under active development • Industrial-strength BPEL workflow enactment engine • Scalable • Persistence • Hot-deployment • Reads BPEL and WSDL files from deployment archive (BPR files) • Listens for incoming messages to trigger deployed processes

15. UCL Department of Computer Science ActiveBPEL Engine Management Console

16. UCL Department of Computer Science ActiveBPEL Monitoring Console

17. UCL Department of Computer Science 2. Introduction to Workflows • Some background about BPEL • Scientific Workflows • Orchestration vs. Choreography

18. UCL Department of Computer Science Workflows • Academic research since 1970’s • Various graphical notations • important impetus for adoption of workflows in business is Enterprise Application Integration (EAI)

19. UCL Department of Computer Science EAI & SOA • EAI refers to the need of integrating a number of different often heterogeneous applications across an enterprise • Simplified by Service Oriented Architecture (SOA) • An application architecture • Separate functions (of the various applications) are loosely coupled • Separate functions can be invoked via well-defined interfaces • SOA promises reusability and flexibility • SOA simplifies the EAI problem • Services can be re-used in ways that were not anticipated at design time • Value of services (applications) can increase by composing services over time • Composition vs. extension

20. UCL Department of Computer Science SOA & BPEL • SOA is an application architecture • Web services are the primary technology for implementing SOAs • Composition of Web services is needed to realise the vision of SOAs • BPEL is the primary industry standard for composing Web services

21. UCL Department of Computer Science Some BPEL Background • BPEL is the result of convergence of two previous workflow languages • XLANG (Microsoft) • WSFL (IBM) • Supported by vast array of providers • IBM, Oracle, Parasoft, Active Endpoints, Microsoft, etc. • BPEL is a reactive workflow language based on events • BPEL makes use of the Web services stack • SOAP, WSDL, UDDI, WS-Addressing, …

22. UCL Department of Computer Science Some BPEL Background (continued) • An XML-based programming language • Provides control and data flow constructs for combining Web services • A BPEL process consists of a set of activities and links between those activities • Expressed in an XML file that can be read and executed by any conformant BPEL engine

23. UCL Department of Computer Science Scientific Workflows • Proliferation of OGSA Grid computing infrastructures • Need to combine grid services into larger services or into experiments • Apply computationally expensive models • Then filter and convert resulting data • Experiments need to be changed frequently to incorporate new insights and ideas • Application scientists need ownership of their workflows

24. UCL Department of Computer Science Scientific Workflows (continued) • BPEL is primarily targeted at business workflows • Scientific workflows differ in a number of ways • The main difference is one of scale along several dimensions

25. UCL Department of Computer Science Orchestration vs. Choreography Taken from http://blog.whatfettle.com/archives/000250.html Orchestration • Central process acts as controller of involved Web services • Explicit definition of control and data flow in central process • Involved Web services have no knowledge of their involvement in higher-level application Choreography • All involved Web services are aware of their partners and when to invoke operations • No central controller • Focus on message exchange

26. UCL Department of Computer Science 3. Fundamental BPEL Concepts • Namespaces • Partners • Variables • Messages

27. UCL Department of Computer Science Namespaces • XML namespaces ensure that XML type definitions are unique (similar to Java package names) • Important when combining services that were created autonomously • A namespace is identified via a globally unique identifier called a URI • A namespace declaration is of the form: • xmlns:<prefix>=“<URI>” • xmlns:test=“http://test.com” • The prefix allows us to use qualified names in XML documents: • <test:name>…</test:name>

28. UCL Department of Computer Science Namespaces (continued) • Each BPEL process defines its own targetNamespace <process name=“HelloWorld" targetNamespace="http://omii-bpel.ac.uk/helloworld" xmlns:tns="http://omii-bpel.ac.uk/helloworld" /> • This serves to qualify the elements of a process (e.g., element declarations, type definitions, messages, portTypes, etc.) • Also allows WSDL definition to be split across several files (one for interface description, another one for implementation description) and use targetNamespace to import parts • In the Sedna editor you can specify the target namespace in the new workflow wizard or on the overview page

29. UCL Department of Computer Science Namespaces (continued) Above: The setup area for namespaces and the target namespace on the overview page. Right: The new workflow wizard allows you to import the process’s target namespace from an existing WSDL definition.

30. UCL Department of Computer Science Partners • We need to define the interactions between a process and its partners • These interactions occur through Web service interfaces • In BPEL this relationship is represented via partner links • In WSDL via partner link types

31. <partnerLinkType name="BuyerSellerLink" xmlns="http://schemas.xmlsoap.org/ws/2003/05/partner-link/"> <role name="Buyer"> <portType name="buy:BuyerPortType"/> </role> <role name="Seller"> <portType name="sell:SellerPortType"/> </role> </partnerLinkType> Each partner link type defines up to 2 roles Each role must support exactly one WSDL portType In this example portTypes for each role have different namespaces In case of asynchronous communication (later) both portTypes can be from same namespace A portType can define exactly 1 role That denotes a service that does not impose any requirements on other service UCL Department of Computer Science Partners – WSDL

32. UCL Department of Computer Science Partners - BPEL • In BPEL we use partner links to define the other participants of an interaction • A partner link has a name used throughout the process to identify a particular partner • Name is used in activities to obtain list of operations to invoke (from portTypes) • Partner link specifies the appropriate partner link type • Has two role attributes: • myRole  role of the process in interaction • partnerRole  role of the partner in interaction

33. UCL Department of Computer Science Partners - BPEL • In Sedna use Partner Wizard on overview page to set up partners • Click Add Partner button for new partners • Click partner icons to modify existing partners • Specify URL of file location of partner’s WSDL definition • Follow guidance given by wizard • Example of partner with just myRole attribute set

34. UCL Department of Computer Science Variables • BPEL variables can be used to store messages or any other kind of data representing state of process • Type of a variable can be • WSDL message type • XML Schema simple type • XML Schema element • An XML Schema complex type must be associated with an element to be used as the type of a BPEL variable • Only variables of a message type can be used as input/output in operations (invocation of operations on service partners) • Types can be defined in some XSD, the process WSDL, or some partner WSDL

35. UCL Department of Computer Science Variables (continued) • Two ways to assign value to variable • Use assign construct • Receipt of message assigned to variable • Can have global variables and local variables defined in scope • Scoping Rules • Local variable overrides outer variable with same name and type • Variable visible in its own scope and all nested scopes • Must not have variables with same name but different types in one enclosing scope hierarchy

36. UCL Department of Computer Science Variables (continued) • In Sedna use overview page to set up global process variables • Click Add Partner button • In wizard • Type variable name • Select message type from drop-down list • Wizard uses set up partners to present list of message types • Does not currently support XSD types directly (will discuss later) • Modify variables by clicking icons

37. UCL Department of Computer Science Messages • WSDL message has a unique name and consists of several parts • Parts are linked to type definitions or element declarations • WSDL types element allows for definition of types, elements, and import of external schemas

38. UCL Department of Computer Science Messages <types> <schema targetNamespace… xmlns=“http:www.w3.org/2001/XMLSchema”> <element name=“FirstName” type=“xsd:string”/> <element name=“Age” type=“xsd:int”/> <complexType name=“employeeType”> <sequence> <element name=“name” ref=“FirstName”/> <element name=“old” ref=“Age”/> </sequence> </complexType> </schema> <schema …> <import namespace="http://data.country.samples.sedna.bpel.omii.ac.uk" schemaLocation="http://localhost:8080/axis/schemas/CCSData.xsd"/> </schema> </types>

39. UCL Department of Computer Science Messages <message name=“EmployeeInputMsg”> <part name=“main” type=“tns:employeeType”/> </message> • Message has a name attribute which must be unique • part name attribute must be unique within message scope • type attribute refers to a type definition • element attribute refers to an element declaration • When should we define a part via a type def and when via an element declaration? • WS-I 1.0 Basic Profile and possibly WSDL 1.2: A message must not contain parts that specify type as well as element attributes

40. UCL Department of Computer Science Messages • Messages form the input and output to operations • Their parts are transformed into SOAP Body element content

41. Assignments Receive Reply Pick Switch Wait Invoke Process Scope Sequence Flows While UCL Department of Computer Science 4. BPEL Language Elements

42. UCL Department of Computer Science Assignments • Assign activity to copy data among variables (from source to target) • Can also use it to construct data • Can calculate value of expression and store in variable • Supports • XPath functions • BPEL functions (extensions to XPath functions) • XQuery • XSLT • Powerful but complex

43. UCL Department of Computer Science Assignments • Type compatibility • From variable of WSDL message type to another variable of WSDL message type  must be the same WSDL message type • Source and target are XML Schema types or elements  source must contain element or type associated with destination

44. UCL Department of Computer Science Assignments – Literal XML • Assign literal XML or any other kind of value to a target variable (e.g., part of a message) • Selecting assign activity in editor populates corresponding properties view

45. <employee> <name>Bruno</name> <age>26</age> </employee> Enter this into from (literal) field Then, select To (container) to specify the target variable and To (part) to specify the part of the message <message name=“CandidateInMsg”> <part name=“payload” type=“tns:employeeType”/> </message> <complexType name=“employeeType”> <sequence> <element name=“employee”> <complexType> <sequence> <element name=“name” type=“xsd:string”/> <element name=“age” type=“xsd:int”/> … UCL Department of Computer Science Assignments – Literal XML

46. 26 Enter this into From (Expression) field Then, select To (container) to specify the target variable and To (part) to specify the part of the message Then enter following To (Query): /employee/age <message name=“CandidateInMsg”> <part name=“payload” type=“tns:employeeType”/> </message> <complexType name=“employeeType”> <sequence> <element name=“employee”> <complexType> <sequence> <element name=“name” type=“xsd:string”/> <element name=“age” type=“xsd:int”/> … UCL Department of Computer Science Assignments – Numerical Value + Query

47. UCL Department of Computer Science Assignments – Copying Between Variables • Select names of source and target variables • Select source and/or target variable parts as needed • Can use several assign statements in a row

48. Messages <msg name=“SimpleEmployeeMsg”> <part name=“payload” type=“employeeType”/> </msg> <msg name=“ComplexEmployeeMsg”> <part name=“main” type=“employeeRecordType”/> </msg> Variables <variable name=“source” type=“SimpleEmployeeMsg”/> <variable name=“target” type=“ComplexEmployeeMsg”/> employeeRecordType <complexType name="employeeRecordType"> <sequence> <element name=“employeeRecord”> <complexType> <sequence> <element name="FirstName" type="xsd:string"/> <element name="Confidential"> <complexType> <sequence> <element name="Age" type="xsd:int"/> </sequence> </complexType> </element> … UCL Department of Computer Science Assignments – Copy Between Variables with Query

49. UCL Department of Computer Science Assignments – Copy Between Variables with Query

50. UCL Department of Computer Science Assignments – XPath Functions • getVariableData(‘var_name’,’part_name’,<query>) • Making me a year older: bpws:getVariableData(‘source’,’payload’,’/employee/age’) +1 • Enter this in From (Expression) field • Remaining settings stay as before • Greeting me: concat(‘Hi ‘,bpws:getVariableData(‘source’,’payload’, ‘/employee/name’))