1 / 21

Selim Kalayci, S. Masoud Sadjadi School of Computing and Information Sciences

Pattern-based Decentralization and Run-time Adaptation Framework for Multi-site Workflow Orchestrations. Selim Kalayci, S. Masoud Sadjadi School of Computing and Information Sciences Florida International University.

iren
Download Presentation

Selim Kalayci, S. Masoud Sadjadi School of Computing and Information Sciences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pattern-based Decentralization and Run-time Adaptation Framework for Multi-site Workflow Orchestrations Selim Kalayci, S. Masoud Sadjadi School of Computing and Information Sciences Florida International University SEKE 2013: The 25th International Conference on Software Engineering and Knowledge Engineering

  2. Background • Scientific workflows capture the business logic of complex applications in various fields • Directed Acyclic Graphs (DAGs) are commonly used to represent them • Basic Lifecycle of workflows: Abstract Workflow Concrete Workflow Workflow Orchestration SEKE 2013

  3. Workflow Orchestration • Orchestration of workflows that span multiple site of resources. • e.g., XSEDE, national and international Grid infrastructures • Heterogeneous and dynamic computing environments • Centralized Orchestration Issues • Scalability • Additional overhead • Non-optimal Adaptation decisions, due to lack of detailed resource information SEKE 2013

  4. Contributions of this Paper • Decentralized orchestration of workflows • Pattern-based generic framework • Utilizes common DAG patterns (next slide) • Run-time adaptation of workflow orchestration • Pattern-based, non-intrusive framework • Complies with ‘separation of concerns’ • Prototype Implementation based on Condor DAGMan SEKE 2013

  5. DAG Patterns • Building blocks for most types of scientific workflows • High-level representation to describe ‘tasks’ and ‘data/control dependencies’ among them SEKE 2013

  6. Decentralization Framework • Local workflow execution managers collaborate to orchestrate the whole workflow • Increasing scalability • Higher autonomy for local sites • Workflow specification goes through a transformation process • Each site performs locally • Pattern-based in accordance with each mapping scenario SEKE 2013

  7. Transformations for the Sequence DAG Pattern - 1 (a) (b) SEKE 2013

  8. Transformations for the Sequence DAG Pattern - 2 (c) (d) SEKE 2013

  9. Transformations for the Fork/Branch DAG Pattern - 1 (a) SEKE 2013

  10. Transformations for the Fork/Branch DAG Pattern - 2 (b) SEKE 2013

  11. Transformations for the Join DAG Pattern - 1 (a) SEKE 2013

  12. Transformations for the Join DAG Pattern - 2 (b) SEKE 2013

  13. Run-time Adaptation • Problem: dynamic changes in the run-time environment • Solution: • Step 1 – Monitoring and Planning • Step 2 – Enactment of Adaptation Plan • Enactment of Adaptation Plan: basically, modifying the mapping of certain tasks. SEKE 2013

  14. Run-time Adaptation Framework • Our Framework: • Follows the Decentralized approach • Based on DAG Adaptation patterns • Low-level of intrusiveness • At the Originating Site: • Transformation based on DAG Adaptation patterns • At the Destination Site(s): • Capture the transferred tasks and compose the corresponding ‘patch DAG’ • Orchestrate the patch DAG in isolation SEKE 2013

  15. DAG Adaptation for the Sequence DAG Pattern - 1 (a) SEKE 2013

  16. DAG Adaptation for the Sequence DAG Pattern - 2 (b) SEKE 2013

  17. Patch DAG corresponding to the Adapted Sequence Pattern - 1 (a) SEKE 2013

  18. Patch DAG corresponding to the Adapted Sequence Pattern - 2 (b) SEKE 2013

  19. Prototype Implementation • Based on Condor DAGMan • Centralized • No native run-time support • Decentralization • Local Condor DAGMan deployed at each site • Transformations achieved through Pre/Post scripts and lightweight sync tasks • Run-time Adaptation • At originating site: utilize rescue DAGs to keep track of pre/post-adaptation specifications • At destination site: compose and then orchestrate patch DAG specification SEKE 2013

  20. Related Work • Centralized Workflow Execution Managers: • Pegasus (underlying engine: Condor DAGMan) • Taverna • GrADS • ASKALON: hierarchical workflow management system, one master and multiple slave engines • Several studies for decentralization of business processes enacted via BPEL execution engine • Most of the existing Adaptation mechanisms are highly-intrusive SEKE 2013

  21. Thank you! • This material is based upon work supported by the National Science Foundation under Grant Nos. OISE-0730065 and HRD-0833093. • Contact: Selim Kalayci, S. Masoud Sadjadi {skala001, sadjadi}@cs.fiu.edu School of Computing and Information Sciences Florida International University SEKE 2013

More Related