Introduction to Grids

1. Introduction to Grids David Boyd Information Technology Department Rutherford Appleton Laboratory D.R.S.Boyd@rl.ac.uk

2. The Grid - what is it? • “The grid is an emerging infrastructure that will fundamentally change the way we think about - and use - computing. The grid will connect multiple regional and national computational grids to create a universal source of computing power.” (c.f. electric power grid)The Grid: Blueprint for a New Computing Infrastructure, Foster and Kesselman, Morgan Kaufmann Publishers

3. Promise of ubiquitous computing? • “Global virtual computer” • Grid “middleware” as the OS of this computer • Accessible via a “plug on the wall” • Dependable - predictable, sustained performance • Consistent - standard interfaces, services and operating parameters • Pervasive - widely available via controlled access • Inexpensive - perceived cost relative to alternative solutionsWe will eventually take the Grid for granted

4. Where the Grid is coming from . . . • Work over the last decade on several distinct topics • Metacomputing - combining distributed heterogeneous computing resources on a single problem • Data Archiving - building collections of data on specific topics with open metadata catalogues and well-documented data formats • Collaborative Working - network-based facilities for distributed, concurrent working on shared information • Data visualization - large-scale immersive facilities for 3D visual interaction with data coupled to computational analysis • Network-based information systems - WAIS, Gopher, WWW • Instrumentation control - remote control of experimental equipment and real time data gathering systems

5. The Grid (potentially) offers . . . • A consistent way of combining many types of “computing” resource located anywhere to solve complex problems • These resources can be: • networks (LAN, SJ4, Geant(EU), Internet2(US), . . . ) • computing systems (HPC, MPP, SMP, clusters, workstations, PCs, . .) • data storage facilities (tape robots, disk farms, local caches, . . . ) • visualization / VR environments (VR Centres, CAVEs , . . . , desktop) • scientific instruments (telescopes, microscopes, synchrotrons, satellites, . . . ) • . . . . basically any device which can communicate over the internet

6. But the Grid is not yet . . . • mature, reliable technology • a well-understood system • a generic solution • easy to use • capable of sustained operation • economically validated. . . however . . . • I might have said the same about the Web ~7 years ago • and it has since transformed the world’s access to information and created a business revolution . . . imagine the potential of the Grid over the next 5-7 years

7. The Web vs The Grid • The Web supports wide area data/information location and retrieval • The Grid supports complete process initiation and execution including any necessary data location and retrieval • offers the potential to carry out significantly large tasks • opens up new capabilities for knowledge generation • Currently Grid tools provide a relatively low level of operational control • higher level tools will be developed to automate low level processes • agent technology will eventually support real time dynamic process optimisation

8. Process optimisation • By looking at whole processes it will be possible to optimise them using intelligent management strategies • identifying appropriate (by user-specified criteria) computing resources for the task • locating appropriate sources of the relevant data • extracting relevant subsets of this data • assessing options for migrating the data or the task • evaluating the available network service quality • making a decision on a strategy to carry out the user’s task • adapting this to changing circumstances in real time

9. Grid architecture High-energy physics data analysis Collaborative engineering On-line experiments Applications Climate modelling Parameter studies High throughput Collaborative working Remote control Application Toolkits Data intensive computing Remote visualization Directory service Authentication Data access Grid Services Metadata Resource management Fault detection Grid Fabric Archives Display devices Networks Protocols Instrumentation Control interfaces Computers

10. Grid tools and toolkits • There are many useful tools and toolkits now available • e.g. Globus, Condor, Legion, SRB, LDAP, OOFS, . . . • Globus provides resource allocation and management (GRAM), information access (MDS), authentication (GSI) • Condor provides high throughput computing on distributed networks of workstations with checkpointing • Legion is an object-based large scale distributed computing environment designed to handle trillions of objects • Storage Resource Broker provides facilities for managing a distributed data repository which includes multiple copies

11. An application of the Grid Online Instruments The Globus project Advanced Photon Source wide-area dissemination desktop & VR clients with shared controls real-time collection archival storage tomographic reconstruction

12. Science-driven Grid applications • Environmental science • coupled atmosphere and ocean simulations with long simulated timescales at high resolution • Biological science • multiple protein folding simulations to generate statistically valid models of complex molecules • Astronomy - “Virtual Observatory” • searching across many instrument-specific data archives to study a new class of object at all wavelengths • Materials science • combining and analysing data from different experimental facilities to derive the structure of complex new materials

13. CERN – Tier 0 5 Tier model 2.5 Gbps IN2P3 622 Mbps RAL FNAL Tier 1 155 mbps 155 mbps 622 Mbps Uni n Lab a Tier2 Uni b Lab c   Department  Desktop Large Hadron Collider Data Grid

14. Some US Grid initiatives • Grid Physics Network (GriPhyN) a consortium of universities, was recently given $11.8m by NSF over 5 years for projects in particle physics and astronomy • Particle Physics Data Grid is a consortium of major US government laboratories collaborating on Grid developments for the US particle physics programme • NASA Information Power Grid is being built to support coupled multi-disciplinary simulations and to provide a national resource for rapid response crisis and disaster management • Grid Forum is an open forum of the major Grid development teams with Working Groups on high priority problem areas and a remit to develop standards

15. EU DataGrid project • Will link national Grid projects to create a European Grid infrastructure • Strong focus on developing middleware tools • Application-driven from 3 areas • Particle Physics • Earth Observation • Biosciences • 21 partners! (6 main, 15 associated) • Close collaboration with US initiatives • Likely to lead to several follow-on projects to exploit this infrastructure for social and commercial applications

16. CLRC e-Science programme • Project-driven programme to exploit Grid technologies for the benefit of the CLRC science programme and its users by enhancing the CLRC infrastructure and developing Grid expertise • Data Portal - developing a metadata-driven access mechanism for a wide range of scientific data from CLRC facilities and programmes • ATLAS Datastore- Grid-enabling and enhancing the Datastore to Petabyte capacity • Gbit LAN - building a Gbit internal network in preparation for Grid-based applications and imminent SJ4 connectionGrid reference system - developing and supporting a reference Grid platform as a basis for internal Grid application projects • plus several science application pilot projects

17. UK Grid programme • Proposal to Treasury for £90m as part of SR2000 bid • Awaiting news from OST • Use science as the “storm troopers” to build a national Grid infrastructure • Achieve world-beating science as a result • Give UK business a head start on its competitors by early exposure to the technology • Transfer the expertise gained into the UK commercial software industry to develop a global market

18. The End