1 / 25

Enabling eScience Grid Technologies Today & Tomorrow

Enabling eScience Grid Technologies Today & Tomorrow. Ian Foster Argonne National Laboratory University of Chicago Univa Corporation. eScience [n] : Large-scale science carried out through distributed collaborations—often leveraging access to large-scale data & computing. Overview.

audi
Download Presentation

Enabling eScience Grid Technologies Today & Tomorrow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Enabling eScienceGrid Technologies Today & Tomorrow Ian Foster Argonne National LaboratoryUniversity of ChicagoUniva Corporation eScience [n]: Large-scale science carried out through distributed collaborations—often leveraging access to large-scale data & computing

  2. Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary

  3. Science Today is a Team Sport

  4. NEES: Network for Earthquake Engineering Simulation Links instruments, data, computers, people

  5. A B 1 1 9 9 Shared Distributed Infrastructure The Application-Infrastructure Gap Dynamicand/orDistributedApplications

  6. Bridging the Gap:Grid Technology Users • Service-oriented applications • Wrap applications asservices • Compose applicationsinto workflows • Service-orientedinfrastructure • Provision physicalresources to support application workloads Composition Workflows Invocation ApplnService ApplnService Provisioning

  7. Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary

  8. Tool Tool Reliable File Transfer Uniform interfaces, security mechanisms, Web service transport, monitoring MDS-Index MyProxy DAIS GRAM User Svc User Svc GridFTP Host Env Host Env Grid Technology:Service-Oriented Infrastructure User Application User Application User Application Database Specialized resource Computers Storage

  9. User Applications Custom WSRF Web Services Custom Web Services GT4WSRF Web Services Registry Administration GT4 Container WS-Addressing, WSRF, WS-Notification WSDL, SOAP, WS-Security Web Services:Standards, Tools, Interoperability

  10. SSL/WS-Security with Proxy Certificates Services (running on user’s behalf) Authz Callout Access ComputeCenter Rights CAS or VOMS issuing SAML or X.509 ACs Rights VO MyProxy Local Policy on VO identity or attribute authority Rights’ KCA Authentication & Authorization Users

  11. Index Index The Many Dimensions of Scaling Disk-to-disk onTeraGrid >2 Gbyte/s GridFTP: High-performance, reliable data movement Replica location: LIGO = 40M files, 10 sites

  12. Genome sequence analysis Scaling:Grid2003Workflows Sloan digital sky survey Physics data analysis

  13. STAR: 5 TB transfer (SRM, GridFTP) Earth System Grid: O(100TB) online data NASA/NVO: Mosaics from multiple sources Application Examples Fusion Grid: 1000s of jobs

  14. GlobalCommunity

  15. 68 Belgium 66 Venezuela 66 Romania 64 Indonesia 62 Mexico 61 Turkey 60 Malaysia 58 Portugal 57 Austria 54 Ireland 44 Hong Kong 40 Hungary 38 Egypt 38 Argentina 34 Uruguay 31 Ukraine 29 Slovakia 25 Israel 23 Yugoslavia 23 Iran 22 Bulgaria 22 Uzbekistan 22 Czech Rep. 22 N. Korea 21 Lithuania 21 Croatia 20 Finland 20 New Zealand 17 Nigeria 17 South Africa 16 Jordan 16 Slovenia 16 Afghanistan 15 Denmark 15 Philippines 14 Vanuatu 14 Luxembourg 14 Tunisia 12 Virgin Is. (U.K.) 12 Peru 12 Yemen 11 Norway 11 Algeria 11 Armenia 10 Iceland 9 Zambia 9 Virgin Is. (U.S.) 9 Uganda 9 Bosnia & Herz. 8 Kenya 7 Zimbabwe 7 Saudi Arabia 7 Ecuador 7 Macedonia 6 Bolivia 6 Comoros 6 Zaire 6 Lebanon 5 Puerto Rico 5 Namibia 5 Togo 5 Tajikistan 5 Paraguay 5 Albania 5 Sudan 4 Estonia 4 Camaroon 4 Ghana 4 Tuvalu 4 Costa Rica 4 Cuba 4 UAE 4 Tonga 4 W. Samoa 4 Tanzania 3 Syria 3 Bahamas 3 Ethiopia 3 Mongolia 3 Sri Lanka 3 Wallis & Futuna Is. 3 Belarus 3 Bangladesh 2 Falkland Islands 2 Kuwait 2 Sierra Leone 2 Trinidad & Tobago 2 Guyana 2 American Samoa 2 Andorra 2 Georgia 2 Cook Islands 2 Turkmenistan 2 Gabon 2 The Gambia 2 Kazakhstan 2 Macau 2 Malta 2 Jamaica 2 Latvia 2 Turks & Caicos 1 Bhutan 1 Ascension Island 1 Cyprus 1 Mozambique 1 Tokelau 1 Greenland 1 Nepal 1 Swaziland 1 Iraq 1 Serbia 1 Barbados 1 Cambodia 1 Qatar 1 Saint Vincent 1 Laos 1 San Marino 1 Libya 1 Benin 1 Angola 1 Chad 1 Gibraltar 1 Haiti 1 Guatemala 1 Malawi 1 Equatorial Guinea 1 Palau 1 Bermuda 1 Botswana 1 Suriname 26776 U.S. 2753 China 1318 Japan 1017 India 750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea 306 Taiwan 268 France 241 Canada 211 Viet Nam 211 Spain 202 Russia 187 Pakistan 159 Australia 142 Singapore 131 Greece 119 Colombia 111 Poland 109 Netherlands 107 Thailand 94 Switzerland 77 Chile 74 Sweden 38669 downloads in 2004 from globus.org

  16. 68 Belgium 66 Venezuela 66 Romania 64 Indonesia 62 Mexico 61 Turkey 60 Malaysia 58 Portugal 57 Austria 54 Ireland 44 Hong Kong 40 Hungary 38 Egypt 38 Argentina 34 Uruguay 31 Ukraine 29 Slovakia 25 Israel 23 Yugoslavia 23 Iran 22 Bulgaria 22 Uzbekistan 22 Czech Rep. 22 N. Korea 21 Lithuania 21 Croatia 20 Finland 20 New Zealand 17 Nigeria 17 South Africa 16 Jordan 16 Slovenia 16 Afghanistan 15 Denmark 15 Philippines 14 Vanuatu 14 Luxembourg 14 Tunisia 12 Virgin Is. (U.K.) 12 Peru 12 Yemen 11 Norway 11 Algeria 11 Armenia 10 Iceland 9 Zambia 9 Virgin Is. (U.S.) 9 Uganda 9 Bosnia & Herz. 8 Kenya 7 Zimbabwe 7 Saudi Arabia 7 Ecuador 7 Macedonia 6 Bolivia 6 Comoros 6 Zaire 6 Lebanon 5 Puerto Rico 5 Namibia 5 Togo 5 Tajikistan 5 Paraguay 5 Albania 5 Sudan 4 Estonia 4 Camaroon 4 Ghana 4 Tuvalu 4 Costa Rica 4 Cuba 4 UAE 4 Tonga 4 W. Samoa 4 Tanzania 3 Syria 3 Bahamas 3 Ethiopia 3 Mongolia 3 Sri Lanka 3 Wallis & Futuna Is. 3 Belarus 3 Bangladesh 2 Falkland Islands 2 Kuwait 2 Sierra Leone 2 Trinidad & Tobago 2 Guyana 2 American Samoa 2 Andorra 2 Georgia 2 Cook Islands 2 Turkmenistan 2 Gabon 2 The Gambia 2 Kazakhstan 2 Macau 2 Malta 2 Jamaica 2 Latvia 2 Turks & Caicos 1 Bhutan 1 Ascension Island 1 Cyprus 1 Mozambique 1 Tokelau 1 Greenland 1 Nepal 1 Swaziland 1 Iraq 1 Serbia 1 Barbados 1 Cambodia 1 Qatar 1 Saint Vincent 1 Laos 1 San Marino 1 Libya 1 Benin 1 Angola 1 Chad 1 Gibraltar 1 Haiti 1 Guatemala 1 Malawi 1 Equatorial Guinea 1 Palau 1 Bermuda 1 Botswana 1 Suriname Top 10 26776 U.S. 2753 China 1318 Japan 1017 India 750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea 306 Taiwan 268 France 241 Canada 211 Viet Nam 211 Spain 202 Russia 187 Pakistan 159 Australia 142 Singapore 131 Greece 119 Colombia 111 Poland 109 Netherlands 107 Thailand 94 Switzerland 77 Chile 74 Sweden 26776 U.S. 2753 China 1318 Japan 1017 India 750 U.K. 495 Italy 488 Germany 391 Brazil 328 S. Korea 306 Taiwan 38669 downloads in 2004 from globus.org

  17. Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary

  18. There’s Much More to Do:For Example, U.S. Dept of Energy • Fermi National Accelerator Lab • Tevatron • MIT • Bates Accelerator Center • Plasma Science & Fusion Center • Pacific Northwest National Lab • Environmental Molecular Sciences Lab • Univ. of IL • Electron Microscopy Center for Materials Research • Center for Microanalysis of Materials • Argonne National Lab • Intense Pulsed Neutron Source • Advanced Photon Source • Argonne Tandem Linac Accelerator System • Stanford Linear Accelerator Center • B-Factory • Stanford Synchrotron Radiation Laboratory • Brookhaven • National Lab • Relativistic Heavy Ion Collider • National Synchrotron Light Source • Lawrence Berkeley • National Lab • Advanced Light Source • National Center for Electron Microscopy • National Energy Research Scientific Computing Facility Princeton Plasma Physics Lab • Thomas Jefferson National • Accelerator Facility • Continuous Electron Beam Accelerator Facility • Oak Ridge National Lab • High-Flux Isotope Reactor Surface Modification & Characterization Center • Spallation Neutron Source • (under construction) Sandia Combustion Research Facility General Atomics - DIII-D Tokamak James R. MacDonald Laboratory Los Alamos Neutron Science Center Physics Accelerators Synchrotron Light Sources Neutron Sources Special Purpose Facilities Large Fusion Experiments SC Laboratories SC User Facilities Institutions that Use SC Facilities

  19. There’s Much More to Do:E.g., U.S. Dept of Energy • Goal: Any DOE scientist can access any DOE computer, software, data, instrument • ~25,000 scientists* (vs. ~1000 DOE certs) • ~1000 instruments** (vs. maybe 10 online?) • ~1000 scientific applns** (vs. 2 Fusion services) • ~10 PB of interesting data** (vs. 100TB on ESG) • ~100,000 computers* (vs. ~3000 on Grid3) • Not to mention many external partners I.e., we need to scale by 2-3 orders of magnitude to have DOE-wide impact! * Rough estimate; ** WAG

  20. Domain-dependent Domain-independent Scaling Up:Service-Oriented Science Simulation code Expt design Simulation code Content Expt output Certificate authority Electronic notebook Telepresence monitor Simulation server Services Portal server Data archive Metadata catalog Resources Servers, storage, networks Experimental apparatus

  21. VO ATA VO AZA Establishing Policy Across Virtual Organizations ATA: Attribute Authority AZA: Authorization Authority Subject ATA Resource AZA Subject AZA Resource ATA WS Resource WS-Subject GT4-enabled services such as CAS, VOMS, GridShib are initial steps in this direction

  22. Overview • The need for Grid technology • Grid technology today • Future challenges & directions • Summary

  23. Enabling eScience • eScience methods no longer optional but now vital to scientific competitiveness • We’ve demonstrated feasibility, but now need to scale up • Meanwhile, others are investing heavily • EU: ~$70M/yr for “Grid” infrastructure, tech • UK: ~$60M/yr for eScience apps and tech • Germany, Italy, Netherlands, Japan, China, Korea, Singapore, Australia all have programs  How does the U.S. stay competitive?

  24. Acknowledgments • Carl Kesselman and Steve Tuecke, my long-time Globus co-conspirators • Gregor von Laszewski, Kate Keahey, Jennifer Schopf, Mike Wilde, other Argonne colleagues • Globus Alliance members at Argonne, U.Chicago, USC/ISI, Edinburgh, NCSA, PDC • Miron Livny, U.Wisconsin Condor project, Rick Stevens, Argonne & U.Chicago • Other partners in Grid technology, application, & infrastructure projects • DOE SciDAC program, NSF Middleware Initiative, NASA, IBM for generous support

  25. For More Information • Globus Alliance • www.globus.org • Globus Consortium • www.globusconsortium.com • Global Grid Forum • www.ggf.org • Open Science Grid • www.opensciencegrid.org • Background information • www.mcs.anl.gov/~foster 2nd Edition www.mkp.com/grid2

More Related