1 / 11

Condor BirdBath SOAP Interface to Condor

Develop third-party apps interacting with Condor using SOAP interfaces. Supports various languages and allows job submission to multiple Condor pools based on app semantics. Enhance reliability and robustness through a transaction-based process.

eoneil
Download Presentation

Condor BirdBath SOAP Interface to Condor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Condor BirdBathSOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk

  2. Background

  3. Motivations • Adding Web Service interfaces to Condor daemons • Facilitate the development of third-party applications capable of interacting with Condor (remotely). • E.g. build higher-level application specific scheduler that submits jobs to multiple Condor pools based on application semantics • These can be built using a wide range of languages/SOAP packages • BirdBath has been tested on: • Java (Apache Axis, XSUL) • Python (ZSI) • C# (.Net) • C/C++ (gSOAP) • Condor accessible from platforms where its command-line tools are not supported/installed

  4. Schedd Schedd Schedd The (Hypothetical) Big Picture Site D Condor-G Globus/gridSAM Client SOAP SOAP flocking Site A Site B Site C

  5. Shadow Condor SOAP Interfaces • Now available in Condor 6.7.5 • Obtain resource ClassAds • Obtain all daemon ClassAds • Begin/commit/abort transaction • Submit Job • Obtain Job ClassAds • Get/Send file • Removed/hold/release job Execution machine Execution machine Submission machine(s) Central manager Schedd Collector Negotiator Execution machine Execution machine

  6. Clients for BirdBath • WSDLs available that describe each daemon interface • SOAP libraries (stub code) can be generated automatically from WSDLs Custom Client holdJob() Submission machine(s) SOAP library Schedd

  7. Example: Query Collector • Obtain information (ClassAds) about all resources that have over 512MB RAM collectorLocator = new CondorCollectorLocator(); collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”); classAds = collector.queryStartdAds(“Memory>512”);

  8. Example: Job Submission • Create New Transaction • Create New JobId • Send Files ({input files, binaries}) • Create Job Description • Submit Job • Commit Transaction • Transaction based process • Uses two phase commit to promote reliability and robustness • Files transferred in chunks – Base64 encoded SOAP attachments • DAGMan jobs supported

  9. Client Client <?xml version="1.0" encoding="UTF-8"?> <JobDefinition xmlns="http://…"> <JobDescription> <Application> <Executable>/bin/echo</Executable> <Argument>hello world</Argument> </Application> </JobDescription> </JobDefinition> JSDL JSDL SOAP SOAP GridSAM GridSAM Condor Condor SGE SGE Fork Fork … … OMII Collaboration: gridSAM plugin • Job Submission & Monitoring Web Service that submits to range of resource managers (e.g. Condor, SGE, Globus) • Uses JSDL (Job Submission Description Language) for standardised job description

  10. Our plugin accesses Condor via the SOAP interfaces Condor-WS • Building JSDL to ClassAd conversion libraries ClassAd SOAP Submission machine(s) Schedd gridSAM plugin • Existing Condor plugin uses shell-based submission via command-line tools and monitoring of log files Client JSDL SOAP Middle-tier GridSAM Condor SGE Fork …

  11. Future Work • Bundle plug-in into gridSAM • Also provide separate JSDL to ClassAd conversion library • Extensions to other daemons (what functionality to expose?) • For example, allow checkpoint data to be retrieved through the shadow in order to migrate jobs across pools • Roll out daemons on e-minerals mini-grid

More Related