GridFlow: Workflow Management for Grid Computing

1. GridFlow: Workflow Management for Grid Computing Junwei Cao (曹军威) C&C Research Labs, NEC Europe Ltd., Germany Stephen A. Jarvis and Graham R. Nudd Dept. of Computer Science, Univ. of Warwick, UK Subhash Saini NASA Ames Research Center, USA CCGrid 2003, Tokyo, Japan

2. Outline • Background – Grid Workflow • System Architecture • GridFlow User Portal • Global Grid Workflow Management • Local Grid Sub-workflow Scheduling • Fuzzy Timing Techniques • Summary • Ongoing and Future Work CCGrid 2003, Tokyo, Japan

3. Background – Grid Workflow • Workflow Definition WPDL, BPEL4WS, GSFL, ASCI Grid, … • Workflow Systems WebFlow, Symphony, GridAnt, BPWS4J, TENT, … • Component-based Systems CCA/XCAT, SCIRun, CXML, … • Other Systems Condor DAGMan, UNICORE, MyGrid, GEMSS, GridLab, BioOpera, USC Grid failure handling, … CCGrid 2003, Tokyo, Japan

4. Grid Resources • Grid Resource: A particular grid resource is a high-end computing or storage resource that can be accessed remotely. • Local Grid: A local grid consists of multiple grid resources that belong to one organization. • Global Grid: The global grid includes all grid resources that belong to different organizations within a virtual organization. CCGrid 2003, Tokyo, Japan

5. Grid Tasks • Task: Tasks are the smallest elements in a grid workflow, e.g. MPI & PVM programs. • Sub-workflow: A sub-workflow is a flow of closely related tasks that is to be executed in a predefined sequence on grid resources of a local grid (within one organization). • Workflow: A grid application can be represented as a flow of several different activities, each activity represented by a sub-workflow. CCGrid 2003, Tokyo, Japan

6. Grid Management • Mapping grid workflows to the global grid • Mapping grid sub-workflows to local grids • Mapping grid tasks to grid resources CCGrid 2003, Tokyo, Japan

7. Grid Users GridFlow User Portal Global Grid Information Services (Globus MDS) Workflow Management Performance Services (PACE … ) Resource Management (ARMS) Local Grid Sub-workflow scheduling Resource Scheduling (Titan) Grid Resources System Architecture CCGrid 2003, Tokyo, Japan

8. Source Code Analysis Object Editor Object Library HMCL Compiler PSL Compiler CPU Network (MPI, PVM) Cache (L1, L2) PACE Performance Prediction Application Tools Evaluation Engine Resource Tools CCGrid 2003, Tokyo, Japan

9. Titan Resource Scheduling • Heuristic • Evolutionary • Near-optimal: • Makespan • Idletime • Deadlines CCGrid 2003, Tokyo, Japan

10. A A A A ARMS Grid Management • Agent structure • Communication layer • Decision-making layer • Local management layer • Agent hierarchy • Service advertisement • Service discovery • Agent Capability Tables A User CCGrid 2003, Tokyo, Japan

11. GridFlow User Portal CCGrid 2003, Tokyo, Japan

12. S2 startT = 0 exeT = 3 endT = 3 S2 S4 startT = 5 exeT = 7 endT = 12 S4 S1 startT = 0 exeT = 0 endT = 0 S1 S6 startT = 12 exeT = 0 endT = 12 S6 S3 startT = 0 exeT = 5 endT = 5 S3 S5 startT = 5 exeT = 4 endT = 9 S5 Global Grid Workflow Management / 5 / 5 / 7 / 12 CCGrid 2003, Tokyo, Japan

13. Local Grid Sub-workflow Scheduling Scheduling a flow of tasks onto grid resources within a local grid is very similar to the process that schedules a workflow onto different local grids. There are two challenges: • It is a difficult task to provide an accurate prediction on task/workflow start, execution and end times. • Multiple tasks from different sub-workflows may require the same grid resource at the same time. CCGrid 2003, Tokyo, Japan

14. Fuzzy Timing Techniques • Turning the “prediction accuracy” into a fuzzy concept that is represented using fuzzy numbers. π1(τ)=0.5(0,2,6,7) π2(τ)=(2,4,4,6) CCGrid 2003, Tokyo, Japan

15. Fuzzy Number Operations latest earliest sum min max CCGrid 2003, Tokyo, Japan

16. Resource Conflict Solving I • The start time of a task cannot be configured with the latest end time of its pre-tasks directly, since other tasks exists that may use the same resource at the same time. • A first-come possibly-first-serve policy is adopted. This does not order the conflictive tasks explicitly, but adds some information on degrees of possibilities of task start times. CCGrid 2003, Tokyo, Japan

17. Resource Conflict Solving II • All possible start sequences are considered and are combined to provide an estimation of the end time. CCGrid 2003, Tokyo, Japan

18. Summary • GridFlow is a prototype grid workflow management system, focusing on grid workflow simulation and scheduling. • GridFlow is based on a specific grid resource management infrastructure implemented using agent-based methodologies and performance-driven scheduling technologies. • Making grid workflow management a reality also requires to address general grid computing challenges: openness, standards, security and QoS support. CCGrid 2003, Tokyo, Japan

19. Scanning & Preprocessing Numerical Modeling HPC Simulation Analysis, Diagnosis, Design Target Object Ongoing Work – Applications • Developing grid enabled medical simulation services (GEMSS) using GT3 • Developing grid performance services based on historical information analysis • Developing medical application workflows using BPEL4WS CCGrid 2003, Tokyo, Japan

20. Future Work – Agile Computing • Workflow techniques – one of keys for next generation agile (grid) computing Efficiency (Cheap, Large-scale, Pervasive, Continuous, Massive) Grid Computing P2P Computing Internet Computing Agile (Grid) Computing Cluster Computing HPC Supercomputing Flexibility (Performance, Adaptation, QoS, Individualization) CCGrid 2003, Tokyo, Japan

21. For More Information • http://www.dcs.warwick.ac.uk/~hpsg/ • http://www.ccrl-nece.de/~cao/ • http://www.ccrl-nece.de/gemss/ • http://www.agilecomputing.org/ • mailto:cao@ccrl-nece.de CCGrid 2003, Tokyo, Japan