NeSC Workshop Open Issues in Grid Scheduling Ali Anjomshoaa EPCC, University of Edinburgh

1. |epcc| OverviewofaGridSchedulingParadigm NeSC Workshop Open Issues in Grid Scheduling Ali Anjomshoaa EPCC, University of Edinburgh Tuesday, 21 October 2003

2. Grid Scheduling • Why is scheduling so important for Grids? • In order to make coherent and coordinated use of ubiquitous and heterogeneous Grids, Grid management is essential. • Grid management is not an option. • “Scheduling is all about management.” NeSC Workshop - Open Issues in Grid Scheduling

3. Scheduling is about Management • Grids are complex multivariate environments, which are made up of numerous Grid entities that need to be managed. • These Grid entities fulfill various roles, which places them within the remit of different management environments. • Grid entities and their roles are interdependent on each other and their environments. • The management of these Grid entities is, for the most part, the undertaking of Grid scheduling. NeSC Workshop - Open Issues in Grid Scheduling

4. Scheduling is about Management • Have introduced the concepts of: • Grid entities • Grid entity roles • Grid management environments • There is a mapping: • entity roles ↔ management environments • In addition, there are interdependencies: • Between the Grid entities • Between the Grid entity roles • Between the Grid management environments NeSC Workshop - Open Issues in Grid Scheduling

5. Grid Entities • So what are these Grid entities? • A Grid entity is defined by its roles. • Grid entity roles can include: • A Grid entity can have one or more roles. NeSC Workshop - Open Issues in Grid Scheduling

6. Grid Entity Roles • A Grid entity can fulfill many roles. For example: • it can have both a Job role and a Resource role, • it can have both a Job role and a Workflow role, • or indeed any other combination of roles. • A Grid entity role is defined by a set of role specific attributes. • A Grid entity should be managed within different management environments, depending on the various roles that it fulfills. • Could say that the interest of Grid scheduling is in the role management of the various roles of Grid entities. NeSC Workshop - Open Issues in Grid Scheduling

7. Grid Entity Definition • A Grid entity is defined by its roles. • A Grid entity role is defined by a set of role specific attributes. • “A Grid entity is defined by the sets of role specific attributes of its roles.” • The definition of a Grid entity could be captured in a description document as sets of role attributes. • The description document will then allow the Grid entity to be managed according to its various roles, within a Grid. NeSC Workshop - Open Issues in Grid Scheduling

8. Grid Entity Definition Grid Entity NeSC Workshop - Open Issues in Grid Scheduling

9. Grid Entity Definition Grid Entity Resource Role … Job Role … NeSC Workshop - Open Issues in Grid Scheduling

10. Grid Entity Definition Grid Entity Resource Attribute Set … Job Attribute Set … NeSC Workshop - Open Issues in Grid Scheduling

11. Grid Entity Definition Grid Entity Resource Attribute Set … Job Attribute Set Description Document … NeSC Workshop - Open Issues in Grid Scheduling

12. Summary of Concepts • Previously introduced the concepts of: • Grid entities • Grid entity roles • Grid management environments • Have now introduced two new concepts of: • Grid entity role specific attributes • Grid entity description documents NeSC Workshop - Open Issues in Grid Scheduling

13. Grid Entity Roles and Management • Different roles require different management environments. • “The management environment within which a Grid entity is being managed, will depend on the particular role of that Grid entity which is being managed.” • A role type can not be managed in more than one management environment. • A management environment can manage more than one type of role. NeSC Workshop - Open Issues in Grid Scheduling

14. Grid Entity Roles and Management • Role interdependencies need to be managed also. • This will mean that the boundaries of management environments will need to be crossed. • Common factorisations of methods and protocols, from various management environments, need to be made, in order to allow the management of role interdependencies. NeSC Workshop - Open Issues in Grid Scheduling

15. Grid Management Environments • Resource Management Environment • Computing Resource Management Environment • Database Resource Management Environment • Network Resource Management Environment • Job Management Environment • User Management Environment • Policy Management Environment • Account/Finance Management Environment • Security Management Environment NeSC Workshop - Open Issues in Grid Scheduling

16. Grid Management Environments • A Grid management environment sets the rules for the management of a set of Grid entity role types. • The types of Grid entity roles managed in a particular management environment, can not be managed by any other. NeSC Workshop - Open Issues in Grid Scheduling

17. Grid Entity Roles and Management Grid Entity NeSC Workshop - Open Issues in Grid Scheduling

18. Resource Management Environment Job Management Environment Resource Job Resource Role Job Role Grid Entity Roles and Management Grid Entity NeSC Workshop - Open Issues in Grid Scheduling

19. Grid Entity Roles and Management Resource Management Environment Grid Entity Job Management Environment Job Attribute Set Resource Attribute Set NeSC Workshop - Open Issues in Grid Scheduling

20. Grid Management Environments • Different management environments should allow for custom management models, methods, protocols, languages, and semantics. • However, due to the interdependencies between these management environments, they must also share a common set of languages, semantics, and ontologies in an interoperability layer to allow them to interoperate. • Without this common set of interoperability features, it would not be possible to manage role and environment interdependencies. NeSC Workshop - Open Issues in Grid Scheduling

21. Grid Management Environments Security Resource Policy Job User Account NeSC Workshop - Open Issues in Grid Scheduling

22. Grid Management Environments Security Resource Policy Interoperability Layer Job User Account NeSC Workshop - Open Issues in Grid Scheduling

23. The Result? • What is the advantage of these concepts and abstractions? • Discrete roles and management environments allows them to be developed in a highly independent way. • This framework is easily extendable to include new roles and management environments. • This framework allows role definition and attribute sets to be extended or updated, without breaking the framework. NeSC Workshop - Open Issues in Grid Scheduling

24. Meta-Scheduling • Scheduling should allow the management of roles within their respective management environments. • Meta-scheduling should handle the interdependencies between roles and their environments, both within Grid entities, and between Grid entities. NeSC Workshop - Open Issues in Grid Scheduling

25. Summary • Have introduced the concepts of: • Grid entities • Grid entity roles • Grid entity role specific attributes • Grid entity description documents • Grid management environments • Roles are managed within their respective management environments. • Role interdependencies must be managed. NeSC Workshop - Open Issues in Grid Scheduling

26. Conclusions • Managing discrete Grid entity roles in respective management environments allows for the development of highly independent pieces of the Grid management puzzle. • This framework enables manageable extensions and upgrades of the various pieces of the Grid management puzzle. • This framework gives flexibility in defining Grid entities through the attribution of an extensible set of roles. • This framework allows independent management of discrete roles, while enabling management of role interdependencies through the interoperability layer. • This framework should allow the consolidation of disparate scheduling efforts in various collaborations. NeSC Workshop - Open Issues in Grid Scheduling

27. The Holy Grail! • Interoperable Grid architectures based on heterogeneous infrastructure layers, managing heterogeneous computing environments. • Need common interoperability layers: protocols, languages, semantics, ontologies NeSC Workshop - Open Issues in Grid Scheduling