Concepts of grid computing

1. Concepts of grid computing Guy Warner NeSC Training Team gcw@nesc.ac.uk

2. Acknowledgements • This talk was prepared by Mike Mineter of NeSC and includes slides from previous tutorials and talks delivered by: • Dave Berry, Richard Hopkins, Guy Warner (National e-Science Centre) • the EDG training team • Ian Foster, Argonne National Laboratories • Jeffrey Grethe, SDSC • EGEE colleagues • Mark Baker, The Distributed Systems Group, University of Portsmouth, http://dsg.port.ac.uk/mab • Talks at 3rd EGEE conference by • Kyriakos Baxevanidis,Deputy Head,Unit of Research Infrastructures,European Commission, DG INFSO • Dr Spyros Konidaris, European Commission – DG INFSO Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

3. The Grid Metaphor Mobile Access G R I D M I D D L E W A R E Supercomputer, PC-Cluster Workstation Data-storage, Sensors, Experiments Visualising Internet, networks Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

4. The grid vision • The grid vision is of “Virtual computing” (+ information services to locate computation, storage resources) • Compare: The web: “virtual documents” (+ search engine to locate them) • MOTIVATION: collaboration through sharing resources (and expertise) to expand horizons of • Research • Commerce – engineering, … “the knowledge economy” • Public service – health, environment,… Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

5. Institute A Institute C Institute B Institute D “A grid” • The initial vision: “The Grid” • The present reality: Many “grids” • Each grid is an infrastructure enabling one or more “virtual organisations” to share computing resources • What’s a VO? • People in different organisations seeking to cooperate and share resources across their organisational boundaries • Why establish a Grid? • Share data • Pool computers • Collaborate VO Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

6. Application Software Operating System Disks, Processor, Memory, … The Single Computer • The Operating System enables easy use of • Input devices • Processor • Disks • Display • Any other attached devices Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

7. Application Software Middlewarefor sharing computers, servers, printers, … Operating System on each computer Resources connected by a LAN Resources on a Local Area Network User just perceives “shared resources”, with no regard to location in the organisation: - Authenticated by username / password - Authorised to use own files,… Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

8. Application Software Interface between app. and grid Grid Middleware: “collective services” Grid Middleware on each resource Operating System on each resource Resources connected by internet Resources on a grid Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

9. A Grid • Grid Middleware on each shared Resource • Local Area Networks • Connected by Internet • Data Storage • (Usually) batch jobs on pools of processors • Users join VO’s • Virtual organisation negotiates with sites to agree access to resources • Distributed services (both people and middleware) enable the grid, allow single sign-on THE INTERNET Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

10. What characterises a grid? • Co-ordinated resource sharing • No centralised point of control • Different administrative domains. • Standard, open, general-purpose protocols and interfaces • NOT specific to an application • EGEE, NGS support multiple VO’s • Delivering non-trivial qualities of service • Co-ordinated to deliver combined services, greater than sum of the individual components • http://www.gridtoday.com/02/0722/100136.html Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

11. The components of a Grid • Resources • networking, computers, storage, data, instruments, … • Grid Middleware • the “operating system of the grid” • Operations infrastructure • Run enabling services (people + software) • Virtual Organization management • Procedures for gaining access to resources Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

12. Key concepts • Virtual organisation: people and resources collaborating - across admin, organisational boundaries • Single sign-on • I connect to one machine – some sort of “digital credential” is passed on to any other resource I use, basis of: • Authentication: How do I identify myself to a resource without username/password for each resource I use? • Authorisation: what can I do? Determined by • My membership of VO • VO negotiations with resource providers • Grid middleware runs on each resource • User just perceives “shared resources” with no concern for location or owning organisation Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

13. The first driver: e-Science • What is e-Science? Collaborative science that is made possible by the sharing across the Internet of resources (data, instruments, computation, people’s expertise...) • Often very compute intensive • Often very data intensive (both creating new data and accessing very large data collections) – data deluges from new technologies • Crosses organisational boundaries Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

14. Curation, discovery, re-use of knowledge e-Research e-Science The expanding horizons of grids Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

15. The National Grid Service (NGS) • NGS is a production service • Therefore cannot include latest research prototypes! • ETF recommends what should be deployed • Core sites provide computation and also data services • NGS is evolving • OMII, EGEE, Globus Alliance all have m/w under assessment by the ETF for the NGS • Selected, deployed middleware currently provides “low-level” tools • New deployments will follow soon • New sites and resources being added ! Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

16. Commercial Provider UofD PSRE Leeds Man. RAL Oxford NGS Core Nodes: Host core services, coordinate integration, deployment and support +free to access resources for all VOs. Monitored interfaces + services NGS Partner Sites: Integrated with NGS, some services/resources available for all VOs Monitored interfaces + services NGS Affiliated Sites: Integrated with NGS, support for some VO’s Monitored interfaces (+security etc.) BRISTOL CARDIFF GOSC U of C U of A H P C x U of B C S A R Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

17. EGEE is building a large-scale production grid service to: Underpin research, technology and public service Link with and build on national, regional and international initiatives Foster international cooperation both in the creation and the use of the e-infrastructure EGEE – building e-infrastructure Collaboration Pan-European Grid Operations, Support and training Network infrastructure& Resource centres Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

18. Pilot Added EGEE Communities • Initially supported two communities: High Energy Physics and Bioinformatics • Most VO’s linked to a particular experiment • Additional Communities have since been added: • Geophysics • Earth Observation • Chemistry • Working with other communities • E.g. Digital Libraries Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

19. If “The Grid” vision leads us here… … then where are we now? Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

20. Grids: where are we now? • Many key concepts identified and known • Many grid projects have tested, and benefit from, these • Major efforts now on establishing: • Standards (a slow process) (e.g. Global Grid Forum, http://www.gridforum.org/ ) • Production Grids for multiple VO’s • “Production” = Reliable, sustainable, with commitments to quality of service • In Europe, EGEE • In UK, National Grid Service • In US, Teragrid • One stack of middleware that serves many research (and other!!!) communities • Operational procedures and services (people!, policy,..) • New user communities • … whilst research & development continues Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05

21. Summary of grid computing concepts • Flexible collaboration across multiple administrative domains – sharing data, computers, instruments, application software,.. • Single sign-on to resources in multiple organisations • Authorisation, authentication • Need for people-services as well as middleware services • credential authorities, VO managers, support • Drives are towards • Production services (reliable, sustainable,… – against which research projects can plan with confidence) • In Europe, EGEE • In UK, National Grid Service • Standards • Empowering new user communities Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05