1 / 17

Pegasus: Planning for Execution in Grids

Pegasus: Planning for Execution in Grids. Ewa Deelman, Carl Kesselman, Gaurang Mehta, Gurmeet Singh, Karan Vahi Information Sciences Institute University of Southern California. Pegasus Acknowledgement.

harrison-glass
Download Presentation

Pegasus: Planning for Execution in Grids

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pegasus: Planning for Execution in Grids Ewa Deelman, Carl Kesselman, Gaurang Mehta, Gurmeet Singh, Karan Vahi Information Sciences Institute University of Southern California

  2. Pegasus Acknowledgement • Ewa Deelman, Carl Kesselman, Saurabh Khurana, Gaurang Mehta, Sonal Patil, Gurmeet Singh, Mei-Hui Su, Karan Vahi (ISI) • James Blythe, Yolanda Gil (ISI) • http://pegasus.isi.edu • Research funded as part of the NSF GriPhyN, NVO and SCEC projects.

  3. Grid Applications • Increasing in the level of complexity • Use of individual application components • Reuse of individual intermediate data products • Description of Data Products using Metadata Attributes • Execution environment is complex and very dynamic • Resources come and go • Data is replicated • Components can be found at various locations or staged in on demand • Separation between • the application description • the actual execution description

  4. Abstract Workflow Generation Concrete Workflow Generation

  5. Pegasus:Planning for Execution in Grids • Maps from abstract to concrete workflow • Algorithmic and AI based techniques • Automatically locates physical locations for both components (transformations) and data • Use Globus RLS and the Transformation Catalog • Finds appropriate resources to execute • via Globus MDS • Reuses existing data products where applicable • Publishes newly derived data products • Chimera virtual data catalog

  6. Chimera is developed at ANL By I. Foster, M. Wilde, and J. Voeckler

  7. LIGO Scientific Collaboration • Continuous gravitational waves are expected to be produced by a variety of celestial objects • Only a small fraction of potential sources are known • Need to perform blind searches, scanning the regions of the sky where we have no a priori information of the presence of a source • Wide area, wide frequency searches • Search is performed for potential sources of continuous periodic waves near the Galactic Center and the galactic core • The search is very compute and data intensive • LSC is using the occasion of SC2003 to initiate a month-long production run with science data collected during 8 weeks in the Spring of 2003

  8. Additional resources used: Grid3 iVDGL resources Thanks to everyone involved in standing up the tested and contributing the resources!

  9. LIGO Acknowledgements • Bruce Allen, Scott Koranda, Brian Moe, Xavier Siemens, University of Wisconsin Milwaukee, USA • Stuart Anderson, Kent Blackburn, Albert Lazzarini, Dan Kozak, Hari Pulapaka, Peter Shawhan, Caltech, USA • Steffen Grunewald, Yousuke Itoh, Maria Alessandra Papa, Albert Einstein Institute, Germany • Many Others involved in the Testbed • www.ligo.caltech.edu • www.lsc- group.phys.uwm.edu/lscdatagrid/ • http://pandora.aei.mpg.de/merlin/ • LIGO Laboratory operates under NSF cooperative agreement PHY-0107417

  10. Montage • Montage (NASA and NVO) • Deliver science-grade custom mosaics on demand • Produce mosaics from a wide range of data sources (possibly in different spectra) • User-specified parameters of projection, coordinates, size, rotation and spatial sampling. Mosaic created by Pegasus based Montage from a run of the M101 galaxy images on the Teragrid.

  11. Small Montage Workflow

  12. Montage Acknowledgments • Bruce Berriman, John Good, Anastasia Laity, Caltech/IPAC • Joseph C. Jacob, Daniel S. Katz, JPL • http://montage.ipac. caltech.edu/ • Testbed for Montage: Condor pools at USC/ISI, UW Madison, and Teragrid resources at NCSA, PSC, and SDSC. Montage is funded by the National Aeronautics and Space Administration's Earth Science Technology Office, Computational Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.

  13. Current System

  14. Just In-time planning • Partition Abstract workflow into partial workflows

  15. Meta-DAGMan

  16. Other Applications Using Pegasus • Other GriPhyN applications: • High-energy physics: Atlas, CMS (many) • Astronomy: SDSS (Fermi Lab, ANL) • Astronomy: • Galaxy Morphology (NCSA, JHU, Fermi, many others, NVO-funded) • Biology • BLAST (ANL, PDQ-funded) • Neuroscience • Tomography (SDSC, NIH-funded) • http://pegasus.isi.edu • Funding by NSF GriPhyN, NSF NVO, NIH

More Related