1 / 12

Use of Condor by the LIGO Scientific Collaboration

The L aser I nterferometer G ravitational-Wave O bservatory. Use of Condor by the LIGO Scientific Collaboration. Gregory Mendell, LIGO Hanford Observatory On behalf of the LIGO Scientific Collaboration. Supported by the United States National Science Foundation.

cleta
Download Presentation

Use of Condor by the LIGO Scientific Collaboration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Laser Interferometer Gravitational-Wave Observatory Use of Condor by the LIGO Scientific Collaboration Gregory Mendell, LIGO Hanford Observatory On behalf of the LIGO Scientific Collaboration Supported by the United States National Science Foundation LIGO-G080280-00-W

  2. Sources Of Gravitational Waves Black Holes Dense Stars Stochastic Background Supernovae LIGO-G080280-00-W Photos: http://antwrp.gsfc.nasa.gov; http://imagine.gsfc.nasa.gov

  3. Worldwide Interferometers GEO Virgo TAMA LIGO AIGO LIGO-G080280-00-W

  4. Worldwide Data Analysis using The LIGO Data Grid (LDG) LSC: Cardiff LDAS: MIT LSC: Birmingham LDAS: LHO LSC: Golm LDAS: CIT LSC: SYR LDAS: LLO LSC: UWM LSC: PSU 3000+ CPUs with a typical clock speed of 2.6 GHz LIGO-G080280-00-W

  5. The LIGO Data Grid LDG Client/Server Distribution packageName( 'Server' ) version( '4.5' ) pacmanVersionGE('3.18.5') package( 'Server-Environment' ) package( 'VDT_CACHE:Globus' ) package( 'VDT_CACHE:CA-Certificates' ) package( 'VDT_CACHE:CA-Certificates-Updater' ) package( 'VDT_CACHE:Condor' ) package( 'VDT_CACHE:GSIOpenSSH' ) package( 'VDT_CACHE:KX509' ) package( 'VDT_CACHE:MyProxy' ) package( 'VDT_CACHE:UberFTP' ) package( 'VDT_CACHE:EDG-Make-Gridmap' ) package( 'VDT_CACHE:Globus-RLS') package( 'VDT_CACHE:Globus-Core') package( 'VDT_CACHE:Globus-Condor-Setup' ) package( 'VDT_CACHE:PyGlobus' ) package( 'VDT_CACHE:PyGlobusURLCopy' ) package( 'VDT_CACHE:Pegasus' ) package( 'VDT_CACHE:VOMS-Client' ) package( 'VDT_CACHE:Globus-WS' ) package( 'VDT_CACHE:Tomcat-5.5' ) package( 'VDT_CACHE:TclGlobus' ) package( 'Server-FixSSH' ) package( 'Server-RLS-Python-Client' ) package( 'Server-Cert-Util' ) package( 'Server-LSC-CA' ) LIGO-G080280-00-W

  6. The LIGO Data Grid • Users • 500+ scientist in the LIGO Scientific Collaboration • 200+ doing data analysis on the LIGO Data Grid • Grid middleware • LDG Client/Server • Virtual Data Toolkit (VDT) • Globus Toolkit • GSI and X.509 certificates • pyGlobus, tclGlobus, Pegasus, etc… • In house packages • Glue: LSC Data Location & Pipeline Tools • LDR: LSC Lightweight Data Replication • GridFTP for moving data and files • Replica Location Service (RLS) • Onaysis: LSC Online Analysis System • High throughput computing • Condor for most analyses • BOINC for Einstein@Home • LSC Analysis Software • LAL, Matapps, • DMT, etc.. LIGO-G080280-00-W

  7. Use of Condor by the LIGO Scientific Collaboration • Condor handles 10’s of millions of jobs per year running on the LDG, and up to 500k jobs per DAG. • Condor standard universe checking pointing widely used, saving us from having to manage this. • At Caltech, 30 million jobs processed using 22.8 million CPU hrs. on 1324 CPUs in last 30 months. • For example, to search 1 yr. of datafor GWs from the inspiral of binary neutron star and black hole systems takes ~2 million jobs, and months to run on several thousand ~2.6 GHz nodes. LIGO-G080280-00-W

  8. Inspiral Analysis Pipeline • Multidetector pipeline • Pipeline topology is same for all inspiral searches • BNS, PBH, BBH, spinning BBH • Different template/filtering code used for different searches • Can be used for LIGO-GEO and LIGO-VIRGO analysis • Pipeline Description • Inspiral search run on each IFO • Look for triggers coincident in time and mass between detetors • Follow up with signal-based vetoes • Perform coherent analysis of surviving triggers • Follow up candidate events LIGO-G080280-00-W

  9. Example of a LIGO Inspiral DAG LIGO-G080280-00-W

  10. Example DAG Within Dag DAG: finds data and generates Fourier Tranforms used by other DAGs DAG: outputs spectra of power supply data: LIGO-G080280-00-W

  11. Example Test DAG for Condor Regression Testing • Makes fake data for each detector (same code is used in Monte Carlo simulations). • Run the fully-coherent multi-detector continuous-wave search code, used to search for GWs from rotating neutron stars. • Compares the output with reference data. LIGO-G080280-00-W

  12. The LIGO/Condor Success Story • Condor handles most of our searches and is vital to the success of LIGO. • Condor and LIGO have a biweekly telecon to discuss issues & enhancements. • In approximately the past year, Condor successfully • enhanced scaling to support non-trivial O(1M) node DAGs, • implemented option to prioritize nodes, e.g., depth-first traversal of DAGs, • added categories to limit on number of resource intensive nodes in a DAG, • handling of priorities and staggered start of jobs. • Condor is working on a list of enhancements to, e.g., • speed up of starting DAGs by O(100x), • automate finding of rescue DAGs, e.g., when there are DAGs within DAGs, and merging of sub-DAGs. • Add standard universe support on RHEL/CentOS and Debian. • Condor is compatible with BOINC and can run backfill E@H jobs on the LDG clusters when there are idle cycles. • For the future: • Our offline/online high throughput computing needs will continue to grow. • Online jobs moving towards low latency; need to think about computing needs for realtime detection when Advanced LIGO comes on line. LIGO-G080280-00-W

More Related