1 / 24

OpenDSP - Condor to DRMAA layer (DRMAA to Condor ;-) Krzysztof Kurowski Poznan Supercomputing and Networking Center Pola

OpenDSP - Condor to DRMAA layer (DRMAA to Condor ;-) Krzysztof Kurowski Poznan Supercomputing and Networking Center Poland. European Condor Week , Milan, Italy 2 6-29 June 200 6.

sari
Download Presentation

OpenDSP - Condor to DRMAA layer (DRMAA to Condor ;-) Krzysztof Kurowski Poznan Supercomputing and Networking Center Pola

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OpenDSP - Condor to DRMAA layer (DRMAA to Condor ;-) Krzysztof Kurowski Poznan Supercomputing and Networking Center Poland European Condor Week , Milan, Italy 26-29 June 2006

  2. PSNC was officially established as a non-profit institution affiliated to the Institute of Bioorganic Chemistry Polish Academy of Science in 1993 A few years later, a critical mass has been achieved among PSNC and MANs in Poland… and today: PSNC is an SUNExcellence Center in Portals, New Generation Networks and Grid Technologies and well known worldwide research lab PSNC has been actively involved in 20 EU-funded IST projects PSNC is a technology and knowledge transfer center to academia as well as to business (spin-off, start-up, …) PSNC is an operator of PIONIER: Polish Optical Internet Introduction to PSNC European Condor Week , Milan, Italy 26-29 June 2006

  3. Construction of the broadband optical Internet, Development and verification of pilot services and applications for the Information Society, creating a base for new developments in: eScience, eLearning, eHealth, government and local administration as well as industry and business services, Creation of competitive conditions for active software development for new applications and services in the Information Society. PIONIER Strategic Research Objectives European Condor Week , Milan, Italy 26-29 June 2006

  4. PIONIER: the current status You are here European Condor Week , Milan, Italy 26-29 June 2006

  5. Grid R&D at PSNC European Condor Week , Milan, Italy 26-29 June 2006

  6. Merging Computing Nodes Data flow Data flow Data flow iGrid 2005 GridLabdemo Gridge middleware Displaying Extracting Encoding Extracting Encoding Computing Nodes Computing Nodes Computing Node European Condor Week , Milan, Italy 26-29 June 2006

  7. Virtual Laboratory is a distributed workgroup environment, with the main task of providing a remote access to the various kind of rare and expensive scientific laboratory equipment and distributed computational and data resources. iTV platform (TV/multimedia on demand) enables secure and scalable digital content provisioning, which paves a new path for large scale interactive services deployment used by various eScience communities. eHealth Telemedicine project provides a secure web collaboration platform for surgery and on-line consultation in the hospital treatment process together with mobile services for remote and easy-to-use access to medical databases from mobile devices and PDAs. EU eHealth ACGT project brings together internationally recognized leaders, including PSNC, in their respective fields, with the aim to deliver to the cancer research community an integrated Clinico-Genomic ICT environment enabled by a powerful Grid infrastructure. New Generation Mutimedia Services andPortals European Condor Week , Milan, Italy 26-29 June 2006

  8. New Generation Mutimedia Services andPortals European Condor Week , Milan, Italy 26-29 June 2006

  9. Existing approaches (e.g. Globus) in a relatively simple, script-based and often insecure way try to integrate capabilities provided by Distributed Resource Management (DRM) systems with both end users and administrators applications. DRMAA is a GGF API specification for "the submission and control of jobs to one or more Distributed Resource Management (DRM) systems". Providing a single interface to DRM resources, DRMAA substantially eliminates a problem currently facing commercial software developers - proprietary interfaces which make it difficult to integrate applications with DRM systems. DRMAA substantially accelerates the adoption of direct interfaces to DRM systems, where adoption of proprietary interfaces has traditionally been low. DRMAA provides a standard on which people can create new applications using a "write-once" interfaceand makes grid deployment and management easier for systems administrators. Ideas behind DRMAA European Condor Week , Milan, Italy 26-29 June 2006

  10. DRMAA in DRMs European Condor Week , Milan, Italy 26-29 June 2006

  11. Currently, for each end user a physical local account must be set up in advance to allow him access to operations provided by DRMAA. End user has to login first to a submission or execution DRMs host before any routine from DRMAA is invoked. It is possible to use DRMAA APIs onlylocally. Communication processes between end users' applications using DRMAA are not encrypted and usually separated from the Internet by firewalls and appropriate Intranet configuration. Administrators have limited or no control over DRMAA routines used by end users. What is possible now? European Condor Week , Milan, Italy 26-29 June 2006

  12. Customers and Enterprises needs… Currently, customers main concerns are security and interoperability. This is the reason why I asked about DRMAA and Web Services. I have no doubt that your openDSP will address a lot of our interoperability issues. This is great stuff! Melvin Koh Chee Kian Asia Pacific Science & Technology Center Sun Microsystems Inc. European Condor Week , Milan, Italy 26-29 June 2006

  13. Reference implementations of the DRMAA specification available today (e.g in Sun Grid Engine v6 and Condor) allow much better integration of applications with DRMs, but … In addition to DRMAA APIs, consistent, remote and multi-user access, and policy-based job control over various DRMs are needed. Based on our experiences and best practice efforts, we have taken seriously security (Authentication Authorization Accounting), performance, interoperability, easy of use, and flexibility requirements in order to design a set of modules based on DRMAA and its current implementations to provide multi-user access to operations on jobs in DRMs, (in particular in SGE and Condor). Support for standards accepted and widely used by customers (Web Services, SAML, etc.) in order to allow interoperability between business applications and DRMs. Thus, one of the main motivations for designing and implementing DRMAA Service Provider (DSP) is a lack of such solutions today (existing ones do not meet mentioned requirements). What is still missing? European Condor Week , Milan, Italy 26-29 June 2006

  14. OpenDSP architecture Existing DRMAA routines provided by DRMs Open DRMAA Service Provider (openDSP) and its modules European Condor Week , Milan, Italy 26-29 June 2006

  15. - Built-in support of DRMAA job submission, control, monitoring and error handling over DRMs - Authentication and encryption mechanisms based on SSL or GSI - Successfully tested with external services based on Globus Toolkit 2.4.x, 3.2.x and 4.0 - Authorization callouts and external access control list - External accounting and auditing database available - Stage-in/out operations supported over DIME/MIME attachments - JSDL 1.0 POSIX based job description - Successfully tested with the following resource/cluster management systems (where is Condor?): SGE (SGEv6u4, SGE v6u5, SGEv6u7 recommended) over the following OS systems:     - Linux x86, kernel 2.4, glibc >= 2.2     - Linux AMD64 (Opteron), kernel 2.4, glibc >= 2.3     - Solaris (Sparc) 9  64-bit     - Solaris (x86) 9     - Apple Mac OS/X - Java, .NET and C-based example clients and SDKs available- Distributed under the GPL license Release notes (OpenDSP 0.1) European Condor Week , Milan, Italy 26-29 June 2006

  16. On Jun 21, 2006, at 3:39 PM, Peter Troeger wrote: >> Hi Becky, >> >> I received a big improvement patch for the Condor DRMAA library by a >> research project in Poland. The people are interested when they could >> see the changes in the official Condor release. Is there a timeframe >> for a new DRMAA library version ? >> >> Best regards, >> Peter. That's great! I'll add the patch to the next Condor release. Is the patch currently in SourceForge, or should I obtain it from another location? I'll pull the latest files when I hear from you as to where they are. Thanks, Becky Collaboration with Condor European Condor Week , Milan, Italy 26-29 June 2006

  17. >> The Condor people will provide your patched >> version with Condor 6.7.21.Since 6.7.20 >> isout since yesterday, I expect the next >> version in 4-6weeks at the earliest. Would that >> be enough for you ? Once a new Condor version is released please visit: Collaboration with Condor http://sourceforge.net/projects/opendsp European Condor Week , Milan, Italy 26-29 June 2006

  18. FedStageComputingProvider is a generic job execution, monitoring and management HPC service provider accessible through a well-defined Web Service interface based on DRMAA specification and its reference implementations supported by local cluster/resource management systems, such as Sun Grid Engine, Condor or PBS/Torque. Remote Interfaces: Web Service over SSL, GSI and SSL/SAML2.0 with JSDL 1.0 support AAA Module: Tested and integrated with internal and external AAA modules (FedStage Identity Manager and FedStage Authorization Manager) Distributed Resource Manager: Sun Grid Engine (SGEv6u4, SGE v6u5, SGEv6u7, SGEv6u8), Condor (6.7.21 and higher), PBS/Torque(2.0 and higher) Computing Node: Solaris (Sparc), Solaris (x86), FreeBSD, Linux x86, Linux AMD64, 9/10 Mac X OS, Windows 2000/XP, Available: commercial license and open source under GPL (OpenDSP project) What’s next?FedStage ComputingProvider European Condor Week , Milan, Italy 26-29 June 2006

  19. FedStage: a new wave of HPC solutions? • Done from scratch based on our experiencesand lessons learned • Interoperability, Peformance, Security are the key issues and therefore FedStage AAA-based solutions are of primary importance for the cross-organization sharing of applications, data, and computational services in a collaborative business environment today. • FedStage products follow Liberty Alliance and WS-I • W3C standards:  XML, SOAP • OASIS standards: WS-Security, SAML2.0WS-Notification, WS-Reliability, • GGF standards(recommendations): DRMAA, JSDL • IETF standards: SSL/TLS European Condor Week , Milan, Italy 26-29 June 2006

  20. The convergence of Web Services, Grid technologies and SOA/ESAs that currently occurs among enterprise, media, mobile and Internet domains requires a standard and secure approach for virtualization of IT resources and technology independent interoperability across platforms, services and systems FedStage products are loosely coupled services (SOA) which may be individually useful, or they can be composed to be used in specific business processes. FedStage products may be completely self-contained, or they may depend on the availability of other enterprise services. FedStage products capabilities as well ascommunication and data channels among them and consumers are protected by transport and message level security mechanisms. To deliver the best-quality software to the market we combine innovations with best practices, and thus follow: industry specifications, security standards and recommendations, pure Web Services requirements, WS-* extensions. FedStage: a new wave of HPC solutions? European Condor Week , Milan, Italy 26-29 June 2006

  21. Enterprise Access Management And Control Products: FedStage Identity Manager FedStage Authorization Manager Enterprise Infrastructure Service Providers: FedStage Computing Provider FedStage Storage Provider FedStage Notification ProviderOfficial release: 3Q 2006 European Condor Week , Milan, Italy 26-29 June 2006

  22. How old is this picture? European Condor Week , Milan, Italy 26-29 June 2006

  23. Peter Troeger <peter.troeger@hpi.uni-potsdam.de> Dan Bradley <dan@hep.wisc.edu> Becky Gietzel <bgietzel@cs.wisc.edu> Alan De Smet <adesmet@cs.wisc.edu> R. Kent Wenger <wenger@cs.wisc.edu> Collaboration with Condor European Condor Week , Milan, Italy 26-29 June 2006

  24. Thank you! krzysztof.kurowski@man.poznan.pl European Condor Week , Milan, Italy 26-29 June 2006

More Related