Session: Application Deployment on Grids/eScience

1. The 28th Open Grid ForumMünchen, Germany Session: Application Deployment on Grids/eScience

2. Agenda • The presentation of “Application Deployment on Grids/eScience” from Lanzhou University. (25min.) • Question and Answer. (10min.) • Discussing with the topic of “How to make the developments and the Operations on Grids/eScience easier?” (20min.)

3. Application Deployment on Grids/eScience Dr. YI YANGLanzhou University, Chinayy@lzu.edu.cn

4. Outlines • Background • Our team and Project • Application Deployment Interface Framework (ADIF) • Current work • Further work • Publications

5. Background • Grids technologies are developed sophisticated • To be the infrastructure of eScience • Too many Grids • Different countries • Globus, U.S.A • EGEE, EU • CNGrid, China • CGSP, China • …… • Does this met the objectives of the initial Grid? • Different specifications • OGSI • Webservices • WSRF • Different imple-mentation of WSRF • …… • Different purposes • Bioinformatic • Computer Chemistry • Environment • Physics • ……

6. Background • Objectives of Grid • The usage is as convenient as the electricity to a general customer. • Standard, open, general-purpose protocols and interfaces. • A hardware and software integration. • Sharing the resources on the internet. • Co-operations among distributed areas and organizations. ? ? √ √ ! ?

7. Background • Current Grid • Integration the software and hardware. • Sharing the resources as much as more. • To be as the infrastructures of many applications. • BUT • The sharing resources is flat organized. • The usage of applications is always for special purposes. • The usage of applications is difficult for a general customer. • Why?

8. Background Applications • Current development mode of Grid Applications • Developed by the developers of Grid. • Developers must familiar with both thespecifications of Grid and the domainknowledge of special purposes. • The development and maintenance is hard. • Applications tightly coupled with itsrelated Grid. • The leverage between applications andGrid is not clear. • The migration of applications to other gridsseems unable. Gird

9. background • Idea • The “abstraction” is always an important methodology to improve computer science. • Why we don’t abstract the grid application develop interface (ADVI) to higher levels so that it can be used to develop applications by a domain user or specialist. abstract abstract COMPUTER Assembly Language C/C++/Java abstract abstract Grid Architecture ADVI ??

10. Background Grid Applications • The “deployment” of grid applications on Grid • Abstract the ADVI • Separate the development ofGrid and Grid applications • Make them loosely coupled • Application Deployment Interface Framework (ADIF) Deploy Middleware Deploy Middleware Deploy Middleware Other Grid platform Grid platform

11. Our Team • A single research group, which are composed of three departments of two Universities. • ICCM and SC Lab in Lanzhou University • GCD in Tsinghua University • Group Leader：Prof. Li Lian, lil@lzu.edu.cn • Team Member：Prof. Ruishen Zhang, 2 lectors, 3 Ph.Ds, 8 Masters.

12. Project • Basic Research on Grid Specifications • Supported by National Natural Science Foundation of China. • 12 teams in total who are in charge of different parts of the whole specifications respectively • It is hoped to become the National Grid Specifications in China after finished • Started from Jan. 2008 and will end in Jun. 2010. • Funding is 3 million chinese yuan

14. Application Deployment Interface Framework (ADIF) • Abstract the Application Deployment interface from Application Development interface • It is XML documents, and according to BNF. • Developers just deed to generate these documents by web wizards. • Reference from WSRF、WSDL、JSDL etc.

15. Grid platform and grid application can be developed independently. Here, the basic architecture is the grid platform, and applications in different areas will be built on this basic architecture through the standard, uniform interfaces, which make their surface minimal, seeming like a ‘sandglass’. The Conception of Application Deploy

16. The Advantages of Application Deploy • The grid middleware will be the core of gird applications and the shell of the grid middleware • The grid application will provide an easier and more custom user experiences • The standardizing of grid platform and grid application will be promoted rapidly. • It enhances the scalability and extensibility of grid applications • It also provides the possibility of deploying several applications on alternative grid platforms, especially the heterogeneous grid platforms.

17. Current Work • ADIF v1.0 • A general specification • Four sub specifications • Three levels • Grid application level • Grid applicationdeployment level • Grid architecture level • Four interfaces • Resource management • Workflow management • Logging management • Notification managements

18. A general introduction to ADIF Terminologies and Definitions Introduction about four sub specifications and their relationship. The general specification V1.0

19. Application Deployment Resource Management V1.0 • The grid is to integrate the Internet into a giant super-computer to achieve the comprehensive sharing. • Unfortunately, collections of heterogeneously and geographically distributed resources in grid system are assembled loosely and disorderly. • Only limited resources are useful for individual users or communities. • It is intractable to hunt for the proper resources from such huge system according to user’s requirements. • The specification for discovering intended resources, from a common pool collecting all kinds of resources, becomes more crucial for many users.

20. Application Deployment Resource Management V1.0 • At present, the Resource Management includes: • Resource Deploy Here, resources deploying is allowed, which makes individuation requirements customizing possible. Actually, only a document, called deployList, in XML format, is necessary. • Resource Undeploy Under specified circumstances, some resources in User Resource List MAY become obsolete. In order to make sure the practicability of the User Resource List, users are authorized to give these resources away absolutely according to their wishes. • Resource Destroy Usually, some resources should be discarded from the User Resource List when they have become unavailable. For security, under this operation, the unavailable resources are merely deleted from user logic view, and never destroyed in the grid system.

21. Application Deployment Resource Management V1.0 • Example of Resource Deploy RDD Grid Resources User Resourcest

22. Application Deployment Workflow Management V1.0 • In order to support complex research experiments, distributed resources need to be orchestrated while the application workflow operations being managed under Grid environments. • The grid workflow management system thus should provide sets of tools to facilitate building grid application services by orchestrating web services. • From grid perspective, workflow is a coarsely granular unit which is supervised and controlled by the job management system of grid. The focus of workflow is mainly what to do and how to do. • At present, the user not only regards the logic of the whole processes, but also involves different requirements towards heterogeneous environments to submit their computational jobs.

23. Application Deployment Workflow Management V1.0 • Therefore, a set of standard interfaces is proposed as following: • Definition for the process control structure; • Definition for the workflow designer, which addresses some conventions on the structure of the designer; • The Requirement Document Definition, which provides an XML grammar document to implement the users’ requirements about computational jobs, in order to identify an environment to support reliable executions towards workflow.

24. Application Deployment Workflow Management V1.0

25. Application Deployment Logging Management V1.0 • Logging management plays an important part in the grid platform. • The logging can absolutely record all the information of each operation in the grid platform, which can provide all the necessary information for any possible analyzing of the grid administrator. • From the perspective of users, the information, which should be acquainted, is not only the state information of grid platform, but also the record of the information of personal behaviors in the grid platform. • So it is important to manage and maintain the records of the user’s behaviors in the grid system. However, for each user, what they concern about just is their own relevant information among the grid platform. • So the interfaces to extend and shield the logging of grid SHOULD be provided.

26. Application Deployment Logging Management V1.0 • Therefore, in this document the map relationship of logging, between the application system and the grid, is defined. What’s more, the flexible logging definition is provided, and the personnel operation for the logging model is supported. • the logging for Job Result, • the logging for submitting the Atomic Job, • the logging for submitting the Workflow Job, • the logging for packing the Application for gird platform, • the logging for Destroying Resource in User Resource List, • the logging for user Login in platform • the logging for user Logout from grid platform

27. Application Deployment Logging Management V1.0 • Example of logging deployment RDD User logging Mng. User logging

28. Application Deployment Notification Mechanism V1.0 • There are multitudinous resources in Grid System, but the resources, meeting users’ requirements, MAY be only a small part. • So it turns necessary that the resources COULD be subscribed according to users’ requirements. • However, the resources among the Grid System are changing frequently and dynamically • It turns more urgent to provide an interaction pattern between the user and the grid, which is used to inform the users the current statuses of the grid resource.

29. Application Deployment Notification Mechanism V1.0

30. Current Results • ADIF documents V1.0 • ADIF v1.0 • ADIF-WorkflowManagement v1.0 • ADIF-ResourceManagement v1.0 • ADIF-LoggingManagement v1.0 • ADIF-NotificationMechanism v1.0 • A reference implementation on CGSP （China Grid Support Platform）

31. Future Works • Improve the ADIF • Make the ADIF more implacable • Deploy eSoc on CGSP using ADIF • Interoperation between CGSP and CNGrid using ADIF

32. Publications • Jiazao Lin, Wanbing Shi, Yi Yang，Lian Li，A novel model for grid resources management based on user customization, ChinaGrid Annual Conference, 2008-9-12 [EI检索]. • Wanbing Shi, Yi Yang, Chunyan Zhang, Lian Li，A Study on ADIF Based on e-Science Deployment Architecture, GCC2007 IEEE Computer Society,2007.8 [EI检索]. • Lian Li, Ruishen Zhang, Yongwei Wu, Yi Yang,deploy eSOC on CGSP, Tech. Report. Lanzhou University. • Jiazao Lin, Zhili Zhao, Shoubo Li, Huajian Zhang, Lifen Li, Lian Li, Caihong Li，Application-objected Workflow Management System based on Abstract Service，The 8th international conference on grid and cooperative computing (GCC2009)，2009，8（已发表） • Jiazao Lin, Zhili Zhao, Lei Liu, Huarong Sun, Shoubo Li, Caihong Li, Li Liu, Lian Li, MDMF: A Comprehensive Framework for Managing Large-scale Heterogeneous Data in eSoC Collaborative Environment，SWS2009, 2009,pp:138-143 • Chenyang Zhao, Shoubo Li, Zhiwei Liu, Li Liu, Lian Li. A Publish/Subscribe System based on Distributed Broker Management in e-Science environment. SWS2009, 2009, pp:20-25 • Shoubo Li, Jiazao Lin, Chenyang Zhao, Zhili Zhao, Li Liu, Lian Li. GCP:A Load-balancing Tool in e-Science Environment. SWS2009, 2009. pp:90-95.

33. “Although we can see the limit steps we have made, but we still have a lot of works to do.”

34. Thank YouQuestions & Answers

35. How to make the developments and the Operations on Grids/eScience easier? • Is the idea of ADIF correct? • Does the ADIF useful? • IF it does, How to improve? Suggestions and Advices are welcome • Any Other sub specification? • ……

36. Thank You