1 / 30

Genesis II: Easy-to-use, Standards Based Grid Middleware

“Open Source, OGSA Implementation”. Genesis II: Easy-to-use, Standards Based Grid Middleware. Andrew Grimshaw Genesis II Team University of Virginia. Outline. Background Standards Genesis II Demo Conclusion. Background. Specifications cannot exist in vacuum

creda
Download Presentation

Genesis II: Easy-to-use, Standards Based Grid Middleware

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. “Open Source, OGSA Implementation” Genesis II:Easy-to-use, Standards Based Grid Middleware Andrew Grimshaw Genesis II Team University of Virginia Genesis II “Open Source, OGSA Implementation”

  2. Outline • Background • Standards • Genesis II • Demo • Conclusion Genesis II “Open Source, OGSA Implementation”

  3. Background • Specifications cannot exist in vacuum • Implementations vet Specifications • Implementation experience shows how spec’s interact • Grids have been around for a while, but adoption remains low – why? • Usability • Gap between grid designers and grid users Genesis II “Open Source, OGSA Implementation”

  4. Why have Grids failed to cross the chasm? • Applications programmers are exposed to the complexity of the underlying environment • Failures, data/application management, logging, … • Only the enthusiasts want to live on the bleeding edge • Joe six-pack (biologist | chemist | economist | *) just wants to do their domain research – not do heavy duty hacking • “Activation energy” is too high • Once they have spent the effort to port there is little leverage – applications written for one software stack will not work on other stacks – often they will not even work on different versions of the same software stack • They must suffer the pain all over again • Business needs solutions not technology • HTC solutions (e.g., SGE, LSF, Condor) are easy to use – but not really needed cross-site. • Data sharing solutions are primitive and do not integrate well with the existing infrastructure Genesis II “Open Source, OGSA Implementation”

  5. Genesis II • Users FIRST! • Design the system from the ground up with the overriding mantra that users come first! • Users don’t want to know about grids • Provide a secure, cohesive system in a production system available to users today! • Provide an open source, reference implementation of the OGSA and OGSA-related specifications • Use standards and proto-standards available from the OGF and OGSA to • Provide feedback into the OGF process on various standards based on implementation experience Genesis II “Open Source, OGSA Implementation”

  6. Outline • Background • Standards • Genesis II • Demo • Next Steps Genesis II “Open Source, OGSA Implementation”

  7. Standards Represented in Genesis II Shibboleth OGF WS-Addressing WSDL OGSA JSDL WS-Naming ByteIO And Others! SOAP BES RNS WS-Security XACML SAML Genesis II “Open Source, OGSA Implementation”

  8. RNS • Basic directory service maps strings to EPRs. • Similar to Unix directories (EPR instead of inode) • Can “point” to anything that has an EPR. Genesis II “Open Source, OGSA Implementation”

  9. Pathnames traverse the graph/usr/bin/gcc / usr hosts RNS Directory bin A B C Resource gcc Genesis II “Open Source, OGSA Implementation”

  10. OGSA-ByteIO Genesis II “Open Source, OGSA Implementation”

  11. ByteIO Properties Genesis II “Open Source, OGSA Implementation”

  12. JSDL – Job Submission Description Langauge • XML based - extensible • Widely adopted • Five sections • Job structure • Job Identity • Job application • Job resource • Job data staging • Not intended for human consumption Genesis II “Open Source, OGSA Implementation”

  13. Example JSDL <JobDefinition><JobDescription> <Application><POSIXApplication> <Executable>gnomad</Executable> <Argument>ribo</Argument> </POSIXApplication></Application> <DataStaging> <FileName>ribo.d</FileName> <CreationFlag>overwrite</CreationFlag><DeleteOnTerminate>true</DeleteOnTerminate> <Source><URI>rns:/home/mmm2a/data/ribo.567.x</URI></Source> </DataStaging> <DataStaging> <FileName>tmapfinal</FileName> <CreationFlag>overwrite</CreationFlag><DeleteOnTerminate>true</DeleteOnTerminate> <Target><URI>rns:/home/mmm2a/res/final.567</URI></Source> </DataStaging> </JobDescription></JobDefinition> Genesis II “Open Source, OGSA Implementation”

  14. OGSA-BESBasic Execution Service • Service interface for starting and managing remote compute jobs • JSDL as the job specification language • Implementation is not specified • Queue • Fork/exec • Virtualize • Etc. • Emphasis on Basic • Comes together in the HPC-BP Genesis II “Open Source, OGSA Implementation”

  15. WSI-BSP – Basic Security Profile • Most widely used WS security profile. • Specifies many things, including how and where identity tokens are passed, their schema, and what they should be called. • We support username/password, X.509, and SAML. Genesis II “Open Source, OGSA Implementation”

  16. WS-SecureAddressing • A profile on WS-Addressing and WS-SecurityPolicy • Embeds specified URI’s in EPR metadata that correspond to profiled security policies • Tells client the security policy of the refereced endpoint • Embeds public key into EPR so that the client can communicate securely with the endpoint. Genesis II “Open Source, OGSA Implementation”

  17. WS-Naming • Profile on WS-Addressing EPR • Two components • Endpoint Identifiers • Endpoint Resolution • Dynamic rebinding to support traditional distributed system transparencies: migration, failure, replication, etc. Genesis II “Open Source, OGSA Implementation”

  18. Outline • Background • Standards • Genesis II • Demo • Conclusion Genesis II “Open Source, OGSA Implementation”

  19. Goals • Provide an easy-to-use production Grid system with compute and data Grid capabilities to various partner groups and research projects, • Provide a fully functional Grid framework on which further Grid research could be performed, • To “test drive” the various specifications making their way through various standardization organizations to both vet and better understand those specifications, both in isolation, and together as a whole. Genesis II “Open Source, OGSA Implementation”

  20. Genesis II Features • Ease-of-use – focus on user experience • Standards based – both WS and OGF • Data and compute grid • Open Source – Java/Apache Axis • Robust • Secure Genesis II “Open Source, OGSA Implementation”

  21. User’s First • A large percentage of a grid’s target audience is unable or unwilling to learn new interaction abstractions • Instead of asking the user to adapt to the grid, we should adapt the grid to the user Genesis II “Open Source, OGSA Implementation”

  22. User Abstractions • One of the most ubiquitous user interaction abstractions is the file system • Drag-and-drop • Double Click • /proc filesystem • Named pipes • RNS, ByteIO, and WS-Naming provide the foundation for building these abstractions Genesis II “Open Source, OGSA Implementation”

  23. (Most) everything is a file or directory • Files and directories can be accessed without knowing anything about Grids, Web Services, or anything thanks to FUSE/IFS • FUSE/IFS map the Grid into the file system • BES resources, queues are directories • “ls” to list the jobs, “cat” a job to see its state • “cp” a JSDL file into the directory -> start the job up • A shell script can start jobs by copying • Genesis II containers are directories • “ls” to see the services and porttypes • IDP are files/directories • IS’s are directories • “cp” a query file to IS, creates result • RDBMS’s will be directories • The user can access all of these services without dealing with Web Services!! Genesis II “Open Source, OGSA Implementation”

  24. Genesis II - Data • Grid shell • Usual commands, ls, cat, etc. • FTPd for Windows • FUSE for Linux • IFS for Windows • ExportDir tool • Replicated ExportDir Genesis II “Open Source, OGSA Implementation”

  25. Why FUSE? • FUSE is a per-user proxy to the Grid • Permissions and access based on the user’s credentials • FUSE maps Grid name space (RNS) into local operating system. • The Grid looks like a local file system. Genesis II “Open Source, OGSA Implementation”

  26. Genesis II - Compute • Grid shell • Run, qsub, qdel, • BES resources mapped into the file system • All BES resources implement HPC-BP and HPC-FSE • Genesis queues are workload managers • Both BES containers and queues support the RNS interface • Applications can be run by copying JSDL files into containers • JSDL tool • Application Management Genesis II “Open Source, OGSA Implementation”

  27. Genesis II - Security • WSI-BSP • SSL/TLS and identity token(s) in SOAP header • Authentication via • X.509 based • Uses existing credential infrastructure • Signed SAML assertions (groups realized this way) • Username/token • “login” consists of specifying cert file or authenticating to a WS-Trust STS • Message integrity via both transport and message level encryption and signing (configurable) • Authorization via plug-able module • Currently using access control lists Genesis II “Open Source, OGSA Implementation”

  28. Demonstration • ssh onto Linux machine back home • Windows interface from this machine in Catania • Connected to XCG – Cross Campus Grid Genesis II “Open Source, OGSA Implementation”

  29. Steps • Grid shell • Ls /, /bes-containers, /groups, • Login & cd /home/grimshaw • ls, ls –epr, • Fuse mount • In Linux, ls grid, cd home/grimshaw • Ls gnomad, pictures, WorkPC • Gnomad demo, pictures demo, • JSDL tool • ls around in the queues • drop a bunch of jobs into a queue • Cd grid/queues, ls around • ExportDirTool from my desktop Genesis II “Open Source, OGSA Implementation”

  30. Genesis IITake-away messages • Sufficient body of standards exist with which to build interesting, useful grid systems • Available under Apache 2 licenses • Very active project! • Information and Download Page • http://vcgr.cs.virginia.edu/genesisII • Forum • http://www.cs.virginia.edu/forums/viewforum.php?f=26 Genesis II “Open Source, OGSA Implementation”

More Related