1 / 37

GAT

GAT. Alexander Beck-Ratzka. OFFIS, 23.06.06. GAT: Agenda. Introduction Why a new Grid-API? What GAT offers, what not? Examples Architecture The single API groups File (-stream) management, logical file management, AdvertService management, Job management Adaptors Conclusion.

zena
Download Presentation

GAT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GAT Alexander Beck-Ratzka OFFIS, 23.06.06 GAT

  2. GAT: Agenda • Introduction • Why a new Grid-API? • What GAT offers, what not? • Examples • Architecture • The single API groups • File (-stream) management, logical file management, • AdvertService management, • Job management • Adaptors • Conclusion GAT

  3. Why a new Grid-API? GAT as a uniform API for the access to heterogeneous grid technologies / grid middleware. GAT is only a framework; the real grid operationen must be done by adaptors. GAT offers the possibility for accessing adpaptors. New grid technologies only needs to be coupled to GAT via adaptors -> No changes in the source sode necessary, even not If using new grid technologies. GAT enables a simple access to grid technologies. GAT uses the grid middleware, which is available at the moment, where is should be used, and this by only typing a single statement for GAT_JobSubmit or GATFileCopy... GAT

  4. What GAT offers? (1) GAT is not a new grid middleware. • GAT enables an easy to different grid infrastructures. • GRAM • Condor • Unicore • GridFTP • ... GAT is an OpenSource project. GAT

  5. What GAT offers? (2) • Applications call the GAT-API for grid operations. • Applications hav to be linked against the GAT. • Applications are independend of the underlying grid infrastructure. • GAT Engine loads available adaptors at runtime • During a calls of the GAT-API the GAT engine decides, which adaptor performs the grid operation. • In error case during a „grid operation“, another adaptor will be accessed. • default adaptors offer local operations • grid applications can be compiled, linked and tested without available grid services. • The same application can run in a „complete grid environment“: without a new build. GAT

  6. What GAT offers? (3) The GAT-API doesn‘t change. Changes e.g. in Globus job submit are included in the GAT Globus adaptor. GAT offers reliability. If one grid service is not available, another grid service is used. GAT is much eaysier to install as Globus. GAT offers grid with a minimum effort for the end user. GAT

  7. What GAT not offers? GAT doesn‘t replace any function of a grid middleware. Without an adaptor for a grid middleware GAT is useless for this middleware. GAT has no resource broker. GAT

  8. Examples (Java-GAT) File operations Job submit GAT

  9. File copy: Java-GAT import org.gridlab.gat.*; import org.gridlab.gat.io.File; public class RemoteCopy { public static void main(String[] args) throws Exception { GATContext context = new GATContext(); URI src = new URI(args[0]); URI dest = new URI(args[1]); File file = GAT.createFile(context, src); // create file object file.copy(dest); // and copy it GAT.end(); } } GAT

  10. File Write (Java-GAT) import java.io.PrintWriter; import org.gridlab.gat.GAT.*; import org.gridlab.gat.io.FileOutputStream; public class FileStreamSimple { public static void main(String[] args) { GATContext context = new GATContext(); URI src = null; FileOutputStream stream = null; PrintWriter p; src = new URI(args[0]); stream = GAT.createFileOutputStream(context, null, src); p = new PrintWriter(stream); String toBeStreamed = "hello world\n"; p.println(toBeStreamed); p.close(); GAT.end(); } } GAT

  11. Job Submit (Java-GAT) public class SubmitJobToHost { public static void main(String[] args) { GATContext context = new GATContext(); Preferences prefs = new Preferences(); prefs.put("ResourceBroker.adaptor.name", "globus"); prefs.put("ResourceBroker.jobmanager", "sge"); URI exe = null; File outFile = null; File errFile = null; File pre1 = null; File pre1Dest = null; try { exe = new URI("file:////bin/hostname"); outFile = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl/out")); errFile = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl/err")); pre1 = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl//bin/echo")); pre1Dest = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl//home/rob/my_temp_file")); } catch (Exception e) { System.err.println("error: " + e); System.exit(1); } SoftwareDescription sd = new SoftwareDescription(); sd.setLocation(exe); sd.setStdout(outFile); sd.setStderr(errFile); sd.addPreStagedFile(pre1, pre1Dest); Hashtable hardwareAttributes = new Hashtable(); hardwareAttributes.put("machine.node", "fs0.das2.cs.vu.nl"); ResourceDescription rd = new HardwareResourceDescription( hardwareAttributes); JobDescription jd = null; ResourceBroker broker = null; try { jd = new JobDescription(sd, rd); broker = GAT.createResourceBroker(context, prefs); } catch (Exception e) { System.err.println("Could not create Job description: " + e); System.exit(1); } Job job = null; try { job = broker.submitJob(jd); } catch (Exception e) { System.err.println("submission failed: " + e); e.printStackTrace(); System.exit(1); } GAT

  12. GAT architecture API thin layer; only a frame. Adaptor implements “grid functionality” as “ordered” by the API. Engine mediates betwen API u. adaptor: • the adaptors are called at runtime • error tracing and “Fallback” (default local adaptors) CPI for Implementation of single adaptors. GAT

  13. Application layer Application GAT layer GAT API GAT Engine GAT Adapter GTK4 PBS SGE Globus 2/3.x DRMAA Unicore GAT architecture User Space „Grid“ Space GAT

  14. Implementation (Engine) C version fully implemented C++ wrapper fully implemented Python wrapper fully implemented JAVA version fully implemented GAT

  15. Implementation (Adaptors) C-GAT Globus: • gram, gridftp, RLS, gsiscp, gsissh Unicore: • Job Submit, Job Monitoring DRMAA (Distributed Resource Management Application API) SGE (Sun Grid Engine) PBS (Portable Batch System) GAT

  16. Implementation (Adaptors) Java-GAT Globus: • With Java CoG Kit; Globus 3.y complete, GTK 4 up to now without WebServices. Unicore: • In work SGE PBS GAT

  17. Implementation (Adaptors) Java-GAT and C-GAT Local adpaptors: • ssh, scp, ftp, sftp, • File-Adapter: (cp, mv, read, write, etc...) • Job-Adapter: fork, exec, auch über ssh... GAT

  18. File.copy(dest) FileCPI.copy(dest) Adapter1 copy Adapter2 copy Adapter3 copy File.copy: schematic run through GAT

  19. GAT API overview File operations (Monitoring and events) Resources, jobs (Information exchange) (Utility classes: error handling, security, preferences) GAT

  20. File sub systemGATFile class GATObject GATFile Create Destroy Copy Move Delete IsReadable IsWritable GetLength LastWriteTime GetLocation GAT

  21. File sub systemGATFileStream class GATObject GATFileStream Create Destroy Read Write Seek GAT

  22. File sub systemGATLogicalFile class GATObject GATLogicalFile Create Destroy GetFiles GetLocations Remove RemoveFile AddFile AddLocation Replicate GAT

  23. Advert package Goal: Advertisement of important informations at a central place, e.g. job finished, job rc... Realized via an SQL adaptor in C-GAT. GAT

  24. Advert package GATObject GATAdvertService Add Delete AddMetadata GetMetadata GetAdertisable Find SetPWD GetPWD GAT

  25. GATJob Unschedule CheckPoint CloneJob Migrate Stop GetJobDescription GetState GetJobIDGetNativeID Job managementclasses GATResourceBroker ReserveResource FindResources SubmitJob GAT

  26. Job managementavailable job status GAT

  27. GATSoftwareResourceDescription Executable Arguments Stdin Stdout Stderr Pre-Stage-Files Post-Stage-Files GATJobDescription Job managementstructures GATHardwareResourceDescription Machine type Memory CPU-Time Nodes needed GAT

  28. Job managementexample (PBS adaptor) GATHardwareResourceDescription machine.queue = destination (-q) jobname = jobname (-N) yeo = join (-j) memory.size = mem (-l) file.size = file (-l) cpu.walltime = walltime (-l) cpu.nodes = nodes (-l) GAT

  29. Job managementexample (PBS adaptor) GATSoftwareResourceDescription Stdin = stdin (exec < input) Stdout = stdout (-o) Stderr = stderr (-e) Executable = executable Arguments = arglist[] GAT

  30. #PBS -q new@url.org #PBS -l walltime=12:00:00, \ -mem=1G,file=2G,nodes=8@url.org #PBS -N testjob #PBS -o out.testjob #PBS -e err.testjob #PBS -j eo /bin/prog arg1 arg2 lastarg < input . . Job managementexample (PBS adaptor) Realization in QSUB script machine.queue = new@url.org jobname = TestJob yeo = eo memory.size = 1G file.size = 2G cpu.walltime = 12:00:00 cpu.nodes = 8 stdin = input stdout = out.testjob stderr = err.testjob executable = /bin/prog arglist[0] = arg1 arglist[1] = arg2 arglist[2] = lastarg GAT

  31. Job managementlacks Problem: Defintions in HardwareResourceDescription quite arbitrarily. Solution: Orientation at the JSDL standard. GAT

  32. Job managementlacks Problem: To less JOB status. Solution: Orientation at DRMAA. GAT

  33. GAT future C-GAT is now in maintenance mode Java-GAT is still further developed SAGA (Simple API for Grid Applications) is the upcoming GGF standard. First SAGA engine with GAT wrapper and GTK4 adaptor in autumn 2006 GAT

  34. Usage exampleProC MPA Garching GAT

  35. SAGA To be integrated: GAT, Java-CoG, DRMAA, RealityGrid, JSDL, GridRPC, OSGA-BES, GridCPR, gLite, HDF5 Developed by: GAT, RealityGrid UK Science, OMII Grid UK Science, CCT Louisana, VU Netherlands, NAREGI Japan, Globus/CoG, GGF DRMAA, GGF GridRPC Important: Please remarks needs on adaptors! GAT

  36. GAT users C-GAT Java-GAT SuperScalar (Univ. of Barcelona, ESP) SURA-Grid (63 partners, US) SCOOP project (LSU + 9 partners, US) UCOMS project (LSU + 4 partners, US) Cactus (LSU, US)Clusterix Project (PSNC, Univ. of Krakow, PL) Amolf (Vl-e, NL) Frank Seinstra (UvA, NL)Triana group (Cardiff, UK) MPA in Garching ProC-Workflows on the Grid LSU in Baton Rouge Chemistry project AMOLF NL Fourier Transform Mass Spectrometry (FTMS) Analyse. FTMS data are delivered to the Grid with JavaGAT (ssh, sftp, gridftp). Multimedian Project NL Start of parallel Jobs ZIB Berlin INRIA Frankreich GAT

  37. Links for GAT GAT overall: http://www.gridlab.org/WorkPackages/wp-1/ GAT CVS: cvs.gridlab.org cvsroot: pserver:readonly@cvs.gridlab.org:/cvs/gridlab Password: anon GAT sources: wp-1/Codes GAT documents: wp-1/Documents Only GATEngine: wp-1/Codes/GATEngine Download tarball: http://www.gridlab.org/WorkPackages/wp-1/gatreleases.html http://www.gridlab.org/WorkPackages/wp-1/adaptorreleases.html German GAT Mailing-Liste:GAT@d-grid.de (www.listserv.dfn.de) GAT

More Related