250 likes | 414 Views
Enabling eScience Grid Technologies Today & Tomorrow. Ian Foster Argonne National Laboratory University of Chicago Univa Corporation. eScience [n] : Large-scale science carried out through distributed collaborations—often leveraging access to large-scale data & computing. Overview.
E N D
Enabling eScienceGrid Technologies Today & Tomorrow Ian Foster Argonne National LaboratoryUniversity of ChicagoUniva Corporation eScience [n]: Large-scale science carried out through distributed collaborations—often leveraging access to large-scale data & computing
Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary
NEES: Network for Earthquake Engineering Simulation Links instruments, data, computers, people
A B 1 1 9 9 Shared Distributed Infrastructure The Application-Infrastructure Gap Dynamicand/orDistributedApplications
Bridging the Gap:Grid Technology Users • Service-oriented applications • Wrap applications asservices • Compose applicationsinto workflows • Service-orientedinfrastructure • Provision physicalresources to support application workloads Composition Workflows Invocation ApplnService ApplnService Provisioning
Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary
Tool Tool Reliable File Transfer Uniform interfaces, security mechanisms, Web service transport, monitoring MDS-Index MyProxy DAIS GRAM User Svc User Svc GridFTP Host Env Host Env Grid Technology:Service-Oriented Infrastructure User Application User Application User Application Database Specialized resource Computers Storage
User Applications Custom WSRF Web Services Custom Web Services GT4WSRF Web Services Registry Administration GT4 Container WS-Addressing, WSRF, WS-Notification WSDL, SOAP, WS-Security Web Services:Standards, Tools, Interoperability
SSL/WS-Security with Proxy Certificates Services (running on user’s behalf) Authz Callout Access ComputeCenter Rights CAS or VOMS issuing SAML or X.509 ACs Rights VO MyProxy Local Policy on VO identity or attribute authority Rights’ KCA Authentication & Authorization Users
Index Index The Many Dimensions of Scaling Disk-to-disk onTeraGrid >2 Gbyte/s GridFTP: High-performance, reliable data movement Replica location: LIGO = 40M files, 10 sites
Genome sequence analysis Scaling:Grid2003Workflows Sloan digital sky survey Physics data analysis
STAR: 5 TB transfer (SRM, GridFTP) Earth System Grid: O(100TB) online data NASA/NVO: Mosaics from multiple sources Application Examples Fusion Grid: 1000s of jobs
68 Belgium 66 Venezuela 66 Romania 64 Indonesia 62 Mexico 61 Turkey 60 Malaysia 58 Portugal 57 Austria 54 Ireland 44 Hong Kong 40 Hungary 38 Egypt 38 Argentina 34 Uruguay 31 Ukraine 29 Slovakia 25 Israel 23 Yugoslavia 23 Iran 22 Bulgaria 22 Uzbekistan 22 Czech Rep. 22 N. Korea 21 Lithuania 21 Croatia 20 Finland 20 New Zealand 17 Nigeria 17 South Africa 16 Jordan 16 Slovenia 16 Afghanistan 15 Denmark 15 Philippines 14 Vanuatu 14 Luxembourg 14 Tunisia 12 Virgin Is. (U.K.) 12 Peru 12 Yemen 11 Norway 11 Algeria 11 Armenia 10 Iceland 9 Zambia 9 Virgin Is. (U.S.) 9 Uganda 9 Bosnia & Herz. 8 Kenya 7 Zimbabwe 7 Saudi Arabia 7 Ecuador 7 Macedonia 6 Bolivia 6 Comoros 6 Zaire 6 Lebanon 5 Puerto Rico 5 Namibia 5 Togo 5 Tajikistan 5 Paraguay 5 Albania 5 Sudan 4 Estonia 4 Camaroon 4 Ghana 4 Tuvalu 4 Costa Rica 4 Cuba 4 UAE 4 Tonga 4 W. Samoa 4 Tanzania 3 Syria 3 Bahamas 3 Ethiopia 3 Mongolia 3 Sri Lanka 3 Wallis & Futuna Is. 3 Belarus 3 Bangladesh 2 Falkland Islands 2 Kuwait 2 Sierra Leone 2 Trinidad & Tobago 2 Guyana 2 American Samoa 2 Andorra 2 Georgia 2 Cook Islands 2 Turkmenistan 2 Gabon 2 The Gambia 2 Kazakhstan 2 Macau 2 Malta 2 Jamaica 2 Latvia 2 Turks & Caicos 1 Bhutan 1 Ascension Island 1 Cyprus 1 Mozambique 1 Tokelau 1 Greenland 1 Nepal 1 Swaziland 1 Iraq 1 Serbia 1 Barbados 1 Cambodia 1 Qatar 1 Saint Vincent 1 Laos 1 San Marino 1 Libya 1 Benin 1 Angola 1 Chad 1 Gibraltar 1 Haiti 1 Guatemala 1 Malawi 1 Equatorial Guinea 1 Palau 1 Bermuda 1 Botswana 1 Suriname 26776 U.S. 2753 China 1318 Japan 1017 India 750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea 306 Taiwan 268 France 241 Canada 211 Viet Nam 211 Spain 202 Russia 187 Pakistan 159 Australia 142 Singapore 131 Greece 119 Colombia 111 Poland 109 Netherlands 107 Thailand 94 Switzerland 77 Chile 74 Sweden 38669 downloads in 2004 from globus.org
68 Belgium 66 Venezuela 66 Romania 64 Indonesia 62 Mexico 61 Turkey 60 Malaysia 58 Portugal 57 Austria 54 Ireland 44 Hong Kong 40 Hungary 38 Egypt 38 Argentina 34 Uruguay 31 Ukraine 29 Slovakia 25 Israel 23 Yugoslavia 23 Iran 22 Bulgaria 22 Uzbekistan 22 Czech Rep. 22 N. Korea 21 Lithuania 21 Croatia 20 Finland 20 New Zealand 17 Nigeria 17 South Africa 16 Jordan 16 Slovenia 16 Afghanistan 15 Denmark 15 Philippines 14 Vanuatu 14 Luxembourg 14 Tunisia 12 Virgin Is. (U.K.) 12 Peru 12 Yemen 11 Norway 11 Algeria 11 Armenia 10 Iceland 9 Zambia 9 Virgin Is. (U.S.) 9 Uganda 9 Bosnia & Herz. 8 Kenya 7 Zimbabwe 7 Saudi Arabia 7 Ecuador 7 Macedonia 6 Bolivia 6 Comoros 6 Zaire 6 Lebanon 5 Puerto Rico 5 Namibia 5 Togo 5 Tajikistan 5 Paraguay 5 Albania 5 Sudan 4 Estonia 4 Camaroon 4 Ghana 4 Tuvalu 4 Costa Rica 4 Cuba 4 UAE 4 Tonga 4 W. Samoa 4 Tanzania 3 Syria 3 Bahamas 3 Ethiopia 3 Mongolia 3 Sri Lanka 3 Wallis & Futuna Is. 3 Belarus 3 Bangladesh 2 Falkland Islands 2 Kuwait 2 Sierra Leone 2 Trinidad & Tobago 2 Guyana 2 American Samoa 2 Andorra 2 Georgia 2 Cook Islands 2 Turkmenistan 2 Gabon 2 The Gambia 2 Kazakhstan 2 Macau 2 Malta 2 Jamaica 2 Latvia 2 Turks & Caicos 1 Bhutan 1 Ascension Island 1 Cyprus 1 Mozambique 1 Tokelau 1 Greenland 1 Nepal 1 Swaziland 1 Iraq 1 Serbia 1 Barbados 1 Cambodia 1 Qatar 1 Saint Vincent 1 Laos 1 San Marino 1 Libya 1 Benin 1 Angola 1 Chad 1 Gibraltar 1 Haiti 1 Guatemala 1 Malawi 1 Equatorial Guinea 1 Palau 1 Bermuda 1 Botswana 1 Suriname Top 10 26776 U.S. 2753 China 1318 Japan 1017 India 750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea 306 Taiwan 268 France 241 Canada 211 Viet Nam 211 Spain 202 Russia 187 Pakistan 159 Australia 142 Singapore 131 Greece 119 Colombia 111 Poland 109 Netherlands 107 Thailand 94 Switzerland 77 Chile 74 Sweden 26776 U.S. 2753 China 1318 Japan 1017 India 750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea 306 Taiwan 38669 downloads in 2004 from globus.org
Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary
There’s Much More to Do:For Example, U.S. Dept of Energy • Fermi National Accelerator Lab • Tevatron • MIT • Bates Accelerator Center • Plasma Science & Fusion Center • Pacific Northwest National Lab • Environmental Molecular Sciences Lab • Univ. of IL • Electron Microscopy Center for Materials Research • Center for Microanalysis of Materials • Argonne National Lab • Intense Pulsed Neutron Source • Advanced Photon Source • Argonne Tandem Linac Accelerator System • Stanford Linear Accelerator Center • B-Factory • Stanford Synchrotron Radiation Laboratory • Brookhaven • National Lab • Relativistic Heavy Ion Collider • National Synchrotron Light Source • Lawrence Berkeley • National Lab • Advanced Light Source • National Center for Electron Microscopy • National Energy Research Scientific Computing Facility Princeton Plasma Physics Lab • Thomas Jefferson National • Accelerator Facility • Continuous Electron Beam Accelerator Facility • Oak Ridge National Lab • High-Flux Isotope Reactor Surface Modification & Characterization Center • Spallation Neutron Source • (under construction) Sandia Combustion Research Facility General Atomics - DIII-D Tokamak James R. MacDonald Laboratory Los Alamos Neutron Science Center Physics Accelerators Synchrotron Light Sources Neutron Sources Special Purpose Facilities Large Fusion Experiments SC Laboratories SC User Facilities Institutions that Use SC Facilities
There’s Much More to Do:E.g., U.S. Dept of Energy • Goal: Any DOE scientist can access any DOE computer, software, data, instrument • ~25,000 scientists* (vs. ~1000 DOE certs) • ~1000 instruments** (vs. maybe 10 online?) • ~1000 scientific applns** (vs. 2 Fusion services) • ~10 PB of interesting data** (vs. 100TB on ESG) • ~100,000 computers* (vs. ~3000 on Grid3) • Not to mention many external partners I.e., we need to scale by 2-3 orders of magnitude to have DOE-wide impact! * Rough estimate; ** WAG
Domain-dependent Domain-independent Scaling Up:Service-Oriented Science Simulation code Expt design Simulation code Content Expt output Certificate authority Electronic notebook Telepresence monitor Simulation server Services Portal server Data archive Metadata catalog Resources Servers, storage, networks Experimental apparatus
VO ATA VO AZA Establishing Policy Across Virtual Organizations ATA: Attribute Authority AZA: Authorization Authority Subject ATA Resource AZA Subject AZA Resource ATA WS Resource WS-Subject GT4-enabled services such as CAS, VOMS, GridShib are initial steps in this direction
Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary
Enabling eScience • eScience methods no longer optional but now vital to scientific competitiveness • We’ve demonstrated feasibility, but now need to scale up • Meanwhile, others are investing heavily • EU: ~$70M/yr for “Grid” infrastructure, tech • UK: ~$60M/yr for eScience apps and tech • Germany, Italy, Netherlands, Japan, China, Korea, Singapore, Australia all have programs How does the U.S. stay competitive?
Acknowledgments • Carl Kesselman and Steve Tuecke, my long-time Globus co-conspirators • Gregor von Laszewski, Kate Keahey, Jennifer Schopf, Mike Wilde, other Argonne colleagues • Globus Alliance members at Argonne, U.Chicago, USC/ISI, Edinburgh, NCSA, PDC • Miron Livny, U.Wisconsin Condor project, Rick Stevens, Argonne & U.Chicago • Other partners in Grid technology, application, & infrastructure projects • DOE SciDAC program, NSF Middleware Initiative, NASA, IBM for generous support
For More Information • Globus Alliance • www.globus.org • Globus Consortium • www.globusconsortium.com • Global Grid Forum • www.ggf.org • Open Science Grid • www.opensciencegrid.org • Background information • www.mcs.anl.gov/~foster 2nd Edition www.mkp.com/grid2