NIÑOS Take Five: The Management Infrastructure for Distributed Event-Driven Workflows

1. NIÑOS Take Five: The Management Infrastructure for Distributed Event-Driven Workflows SiddarthGanesan, Young Yoon, Hans-Arno Jacobsen

2. Many Workflow Management Systems • IBM Websphere MQ Workflow • Oracle Business Process Management • Sage ERP X3 • SAP Netweaver Business Process Management • Symantec Workflow These are for centralized workflows

3. New breed of workflows Distributed tasks Distributed orchestration Event-driven execution

4. NIÑOS XML (wsdl,bpel) Workflow BPEL Parser Receive Set of sub/advs Deployer Assign Inject Sub/Adv sets Assign Agent Flow PADRES Pub/Sub ESB WS Gateway Agent Invoke Wait Web Service Flow Agent Reply WS Client Execution done Receive Agent Invoke Agent New instance created Reply Agent Wait Agent HTTP/ SOAP

5. Large-scale Business Processes Vendor Goods selection Goods delivery Dispatch B Packaging Pick-up goods Out-stock B FedEx Delivery Pick up Sale prediction Sign Contract Sale Fill order Determinate plan Process Check order CCC administrate Fill out-stock bill Check stock Manufactory Confirm features Design Fill dispatch bill Determinate plan Control Prototype Out Take Raw materials Execute plan Warehouse Material Out-stock B Pay Credit card Check Assign Audit Process control Make plan Target price Signature Raw Check dealer Check credit Finance Confirm Approval Approval Monitoring Feature selection Print receipt Validate Statistic Monitor Marketing Requirement collection Feedback Affirm order Chart Manufactory Strategy Design Marketing Order Payment

6. Research Efforts • Processing infrastructure design and implementation (NIÑOS)[Li et al, TWeb’10] • Enforcement of declarative SLA through fluid workflow processing engines (eQoSystem) [Jacobsen et al, BPM’10, Muthusamy et al, DEBS’11 (Demo)] • Conflict-free execution [Yoon et al, WWW’11]

7. What’s missing? Management!

8. WFMC Workflow Reference Model “Interface 5” http://www.wfmc.org/reference-model.html.

9. Management Mechanisms • Workflow modification Pause, resume, skip • Discovery Workflow status query • Variable update Resource control, user privilege update

10. Management Mechanisms in Distributed Workflow Task A Task B Task C TaskID = 1TaskAgentID = 1InstanceID = xProcessID = 1 TaskID = 2TaskAgentID = 2InstanceID = xProcessID = 1 TaskID = 3TaskAgentID = 3InstanceID = xProcessID = 1 Management Client ADV Task A Agent Pause/Resumeinstance 1 Skip Task B Fetch status of instance 1 InstanceOwnership SUB Task C Agent Task B Agent CausalRelationship Queue

11. Variable Update Accessing or modifying variables and attributes associated with workflow instances and entire workflows • User management operations • Resource control operations such as set, unset, and modify of concurrency levels • Condition variables • Distributed variable updates using variable agents (TWeb’10)

12. Concurrent Composite Operations Management Client 1 Management Client 2 Pause instance Fetch status of instance Resume instance Amend user privilege Resume instance Time

13. Isolation and Ordering Management Client 1 Pause instance Amend user privilege Management Infrastructure Resume instance Management Client 2 Fetch status of instance Pause instance Fetch status of instance Resume instance Amend user privilege Resume instance Resume instance

14. Operation Agent Manager (OAM) Management Client Queue Length Monitor Management Client Operationrequests . . . Dispatchoperations Management Client Pending Running Conflicting . . . Operating Agent (OA) OA OA Management Cluster Elastic! Contacttaskagents TA Task Agent (TA) TA TA TA Distributed task agents

15. Initial Experiment Setup OA OA MC TA n3 n2 n1 TA TA TA OA OA OAM OA OA

16. Overhead of the Management Infrastructure There was an additional overhead of 50 ms for modification operations when compared to discovery operations

17. Elastic Management Infrastructure With elastic management cluster response time remained constant

18. Dynamic Migration TA MC TA n1 n2 n9 n18 . . . . . . n20 n19 OA OA OA OA OAM

19. Benefit of Dynamic Migration The OAs and the OAM are dynamically relocated based on the workload and the network topology to minimize the average delay.

20. Summary • Introduced runtime control (management) of distributed event-driven workflows for the first time • Observed mechanisms for primitive management operations • Identified the isolation and ordering problems of concurrent composite operations • Devised a distributed management infrastructure that is elastic and fluid

21. Thank You