Use of Workflow Techniques for Grid Management

1. Use of Workflow Techniques for Grid Management Junwei Cao (曹军威) C&C Research Laboratories NEC Europe Ltd. Germany NEC, Tokyo, Japan, May 2003

2. CCRLE • Bonn, Germany • ~20 research staffs • 3 working teams: • Numerical simulation • MPI development • Grid Computing • GEMSS NEC, Tokyo, Japan, May 2003

3. My Previous Experience • PACE: performance prediction of parallel and distributed systems • Titan: prediction based job scheduling on clusters and multiprocessors • ARMS: agent-based resource management for grid computing • GridFlow: workflow management for grid computing (CCGrid 2003) NEC, Tokyo, Japan, May 2003

4. Outline • Background • GEMSS objectives • Medical simulation applications • Service performance prediction • Workflow simulation and scheduling • Dynamic service federation • Summary NEC, Tokyo, Japan, May 2003

5. Grid Workflow Management • Workflow Definition WPDL, BPEL4WS, GSFL, ASCI Grid • Workflow Systems WebFlow, Symphony, GridAnt, BPWS4J, TENT • Component-based Systems CCA/XCAT, SCIRun, CXML • Other Systems Condor DAGMan, UNICORE, USC Grid failure handling NEC, Tokyo, Japan, May 2003

6. GEMSS Objectives • Demonstrate that the grid can improve pre-operative planning & near- real-time surgical support by providing access to advanced simulation and image-processing services. • Build middleware on existing or developing grid technology standards to provide support for authorization, workflow, security and Quality of Service aspects. • Develop, evaluate and validate a test-bed for the GEMSS system, including its deployment in the end-user’s working environment. • Anticipate privacy, security and other legal concerns by examining and incorporating into its grid services the latest laws and EU regulations related to providing medical services over the Internet. NEC, Tokyo, Japan, May 2003

7. Medical Applications Medical simulation supports the optimization of operation procedures and the planning of therapeutic strategies. Maxillo-facial surgery Post- surgery Pre- surgery NEC, Tokyo, Japan, May 2003

8. Image Pre-processing An intensity based algorithm (adaptive fuzzy C-means algorithm) is used to provide good quality segmentations for structures of the human head. Identification of substructures CT Images NEC, Tokyo, Japan, May 2003

9. Numerical Modeling Next to the image processing step follows the geometric modeling of the structuressuitable for Finite Element simulations. Mesh generation with Or without smoothing Mesh manipulation using the halo positioning tool NEC, Tokyo, Japan, May 2003

10. HPC Simulation & Visualization The geometric face change Calculated deformation of the skull NEC, Tokyo, Japan, May 2003

11. The Problem … The application includes the use of a complete chain of tools necessary for the entire process from geometric model generation from scan data (segmentation, mesh generation and mesh manipulation) to computer simulation and visualisation. NEC, Tokyo, Japan, May 2003

12. Application Workflow Image Segmentation Computation intensive / Semi interactive Computation intensive / parallel Mesh Generation Mesh Manipulation Full interactive Finite Element Simulation Computation intensive / parallel Visualization Data intensive / Full interactive NEC, Tokyo, Japan, May 2003

13. Grid Enabling … • Interface definition of each module using the WSDL • Implementing each module as a web service using Apache Axis or a grid service using the GT3 • Definition of the whole process using the BPEL4WS • Using a BPEL4WS engine for service invocation • Applied only to Computation intensive parts NEC, Tokyo, Japan, May 2003

14. Challenges Could you finish the process in 1 hours? – QoS and adaptation support is becoming the most active research topic in grid computing community. • Service performance prediction • Workflow simulation and scheduling • Dynamic service federation NEC, Tokyo, Japan, May 2003

15. Grid Performance Services Building performance services as high-level grid services based on OGSA core and base services: • Managing performance-related data • Defining performance metrics • Developing performance analysis algorithms • Developing new APIs for grid service performance prediction NEC, Tokyo, Japan, May 2003

16. Performance-related Data • Application parameters that have an impact on application performance • System status e.g. CPU load, job queue and network bandwidth • Managing performance-related data using OGSA service data support NEC, Tokyo, Japan, May 2003

17. Performance Metrics & Analysis Algorithms Performance metrics • Execution time • Memory usage • Price, and more … Performance analysis – historic information based • Statistical analysis algorithms • Self organizing mapping • More … NEC, Tokyo, Japan, May 2003

18. S2 startT = 0 exeT = 3 endT = 3 S2 S4 startT = 5 exeT = 7 endT = 12 S4 S1 startT = 0 exeT = 0 endT = 0 S1 S6 startT = 12 exeT = 0 endT = 12 S6 S3 startT = 0 exeT = 5 endT = 5 S3 S5 startT = 5 exeT = 4 endT = 9 S5 Workflow Simulation, Scheduling & Rescheduling / 5 / 5 / 7 / 12 NEC, Tokyo, Japan, May 2003

19. Dynamic Service Federation • Using the BPEL4WS service references to select and assign actual partner service dynamically. • Extending BPEL4WS <partner> with some kind of <candidate> element to indicate candidates of a grid service partner. NEC, Tokyo, Japan, May 2003

20. The Solution … Grid Service (WSDL) Performance Services Service data Application portTypes UDDI WSDL Workflow Simulation Engine Workflow Execution Engine Grid Service (WSDL) Performance Services Service data Application portTypes User NEC, Tokyo, Japan, May 2003

21. Other Aspects • Accuracy of performance prediction • Workflow execution monitoring • Security and legal issues • Grid workflow GUIs • Grid data management • Enabling interactive applications • Using workflow techniques for business process management in GEMSS NEC, Tokyo, Japan, May 2003

22. Summary • Programming models for the grid – workflow specification – a candidate? • QoS support and application adaptability – performance prediction + workflow simulation –> dynamic service federation – a solution? • Medical simulation applications – a right target application of the grid? • AgileGrid: agile computing on business grids – a next generation computing paradigm? NEC, Tokyo, Japan, May 2003