Requirements and Expectations from Workflows Asif Akram e-Science Grid Technology Group

1. Requirements and Expectations from Workflows Asif Akram e-Science Grid Technology Group

2. WOSE Project • Workflow Optimisation for Services in e-Science • EPSRC funded project • In collaboration with Imperial College and Cardiff University • CCLRC is investigating the user requirements • Developing Use Cases based on existing e-Science Application • e-HTPX: An e-Science Resources for High Throughput Protein Crystallography

3. Best Practices for Workflows • Modular Design • Exception Handling • Compensation Mechanism • Adaptive Workflow • Flexible Workflows • Management of Workflow

4. Modular Design • Thorough investigation of different activities • Group similar or related activities • Group activities with minimum side effects • Module components should be scalable • Module components can be replaceable • Reliable to serve as repeatable units • Modules operate as “black boxes” in the overall workflow, with their own variables, computational logic, dependency constraints, and event handlers.

5. Exception Handling Exceptions can be due to • inconsistent data, • divergence of tasks from the workflow model, • unexpected contingencies and • un-modeled changes in the environment. • Type of Exceptions: “expected exceptions” (“variations”) and “unexpected exceptions” • Exception Handlers: Global Exception Handlers, Scoped Exception Handlers and Inline Exception Handlers. • Exception handling should be localized

6. CompensationMechanism • “undoing steps in the business process that have already completed successfully” • Reverse the effects of successful activities in the abandoned workflow • Un-handled Exceptions requires compensation • Compensation behavior must be defined by the services to ensure reversal of a operation. • Synchronous operations (i.e. a single connection request/reply) have short lifetimes and do not require compensation to release resources. • Asynchronous communications requires some sort of compensation (unluckily we have normally RPC style Web Services)

7. Adaptive Workflow • Adaptive nature of a workflow can be static or dynamic depending on whether changes are accommodated at design time or runtime respectively. • Redesign and optimization of a process to gain greater efficiency and effectiveness requires adjustment in the workflows; in fact the final goal is to constantly improve the workflow/ process by evolutionary redefinition. • Modifications break a pre-defined process in an attempt to solve the problem in better way. • Dynamically adaptive workflows adjust at runtime; i.e. due to unexpected results

8. Flexible Workflows • Flexibility is required during the early stages of a workflow design when services are un-stable. • Flexibility in the data types and service endpoint. • Flexibility in data, is achieved by applying generic “base” or “parent” data types • Flexibility in service endpoint, requires integration of additional and re-located services through WS-Addressing. • Potential callouts and logic for selecting partner service has to be built into the business process as external configuration file . • Assuming all the services have “similar” interfaces which may not always be the case.

9. Management of Workflow • Workflow management involves the addition of • Breakpoints arbitrary location crucial to the process • Steering capabilities generalizes breakpoints and involves reset, re-schedule, restart, undo, abort, complete, recover, ignore or jump operations. • Persistence of state for fault tolerance, to allow re-execution with different experimental data sets, and to facilitate inspection of intermediate data values • Monitoring foroptimization and analysis of internal state, data flow, inspection of values, re-scheduling of tasks and re-ordering of tasks

10. Portals & Web Services Service End-points Workflow Internet Repository for Authorisation and Policies Clients

11. Business Process Execution Language • Builds on top of XML and Web Services • Abstract, which allow specification of the public message exchange between participating services and doesn’t include the internal details of process flows. • Executable, which allows specification of the exact details of a workflow. Normally BPEL is used for executable processes. • Can utilize other Web Services specification for reliable messages, transactions etc

12. BPEL • Provides different types of activities • Primitive activities can be structured in Modules • BPEL support different types of structured activities i.e. “sequence”, “flows” and “scope” • Extensive support for Exception handling • Can trigger Events from “Exception Handler” • Compensation Handlers can be called from Exception Handler • Limited monitoring is possible through different primitive activities like “onMessage”, “onAlarm” and “pick”

15. BPEL Extension • Different implementation has varying level of extensions • Oracle BPEL engine supports “notification” • Oracle supports “NFlow” for parallel execution of any module “n times” • Oracle has proprietary management capabilities and user interaction functionality • Engines provides API to query and interact with process at runtime • Proprietary API to populate SOAP Headers • Limited support for WS-Security • Support for WSIF and EJB

16. Limitations of BPEL • Limited reusability of primitive and structured activities • Can’t re-execute activities defined earlier • <onMessage> or <onAlarm> events are not to modify the workflow at runtime • BPEL specification does not explicitly support user interactions and notifications • Difficult to integrate security without support from Vendors • BPEL is fairly complex specification

17. FutureWork • Moving to Document Style Web Services • Evaluating WSRF for e-HTPX • e-HTPX mainly passes URL of images in different services (candidate for Resource Properties) • Moving EPR across services is more efficient • Built in support for Notifications • Support for persistence (flat files but can be extended for DB); e-HTPX is using Hibernate • WS-Core supports XML Database Xindice • Support for different level and type of security