Grid Programming Environment: Implementation & Globus Interface

1. Grid Programming Environment: Implementation & Globus Interface Grid Day in CGSL November 3, 2005 Ralf Ratering Thomas Kentemich Alexander Lukichev • Igor Odintsov

2. Agenda • Intel GPE Component Overview • Intel GPE Implementation • Setting up a GPE Grid • Globus Interface • Current State and Next Steps

3. Intel GPE Component Overview

4. Grid SDK GridBeans Client Framework Application Programming Layer Grid Programming Library Service Oriented Architecture/Infrastructure (SOA & SOI) Grid Middleware GPE Component Overview Examples and Templates Applications FSI, Energy, Mfg, Telco, Pharma, Retail, ... Application Programming Layer Globus Toolkit-based Open Source implementation (GPE4GTK) Management Security Prototype Resource Virtualization Layer ... Operating System Operating System Network Network Compute Storage Compute Storage

5. Virtual Machine Virtual Machine OS OS Target System Interface (TSI) Target System Interface (TSI) Firmware ( Platform S/W) Firmware ( Platform S/W) Platform H/W & Platform *Ts(Compute, Storage, Network) Platform H/W & Platform *Ts(Compute, Storage, Network) = GPE components (not started) = GPE components under development GPE Architecture(Fitting in Grid SW Architecture) Applications Application Programming Layer Application Client Web portals Remote File Manager 3 Grid Application I/F (WSRF, JSDL, ...) Expert Client GridBeans Mgmt Console Grid Middleware Workflow Engine Database Access Scheduler / Broker Admin Client Target System Registry Application Deployment Service Hosting Environments and Server Implement. (UNICORE/GS, GTK4, …) 2 Grid Platform I/F (WSRF, WSDM, JSDL, …) Grid Platform Grid Platform Service OS Target System Factory (TSF) Target System Service (TSS) Job Management Service (JMS) Storage Management Service (SMS) File Import Service (FIS) File Export Service (FES) ... MRTE create & manage ... 1 H/W Platform I/F (CIM & WS–based)

6. Emerging Standards • OGSA (Open Grid Services Architecture) • Share components with other architectures • WS* (WS-Addressing, WSRF, WSN, etc.) • Interoperation with other Grid Middleware • JSDL (Job Submission Description Language) • High level job description that can be submitted to all target systems offering a JSDL interface • BPEL (Business Process Execution Language) • Integration of Grid Bean services into larger business process workflows • CIM (Common Information Model) • Used to describe resources • Usage of CIM management interfaces for Grid administration

7. Intel GPE Implementation

8. Expert Client • Light-weight Java application running on mobile devices • Simple application specific interface Application Client • JSR-168 compliant application portlets that can be integrated into existing portals like UPortal, Jetspeed or GridSphere Portal Client GPE Client Framework • Full access to Grid for expert users and administrators • Workflow Editor • Supports multiple applications and user identities

9. GridBean GridBean GridBean Expert Client GridBean GridBean GridBean GridBean GridBean GridBean GridBean Portal Client Write GridBeans for your applications GridBean SDK GPE Client API, libraries and tools GridBean Service Application Client • A GridBean dynamically integrates application-specific user interfaces in different clients

10. SOAP, WSDL, WSRF, WS-Addressing, WS-Security, JSDL and other specifications Atomic Service Client API Atomic Services GPE as interoperability framework UNICORE/GS Expert Client Globus Toolkit 4 China Grid Support Package Application Client Other OGSA-compliant Grid servers Portal Client

11. Atomic Services Overview Target System Factory (TSF) • Atomic service interfaces define basic set of operations and properties that have to be available on a Grid system • Operations describe the functionality • Properties describe the state (WS-Resource Framework) • Different implementations of interfaces for different infrastructures • Protocols, languages, security models, ... Add a new target system to the Grid Target System Service (TSS) Manage target system Job Management Service (JMS) Manage Jobs on target system Storage Management Service (SMS) Manage files on storage Manage transfers to/from storage File Transfer Service (FTS)

12. Target System Factory (TSF) TSR-EPR create(config) Memory=1Gb Running Jobs=23 CPU Type=ia64 Storage=Temp, Root etc. create Extends WS-RP/LT Job-EPR submit(JSDL, initialTT) Target System Service (TSS) Target System Extends WS-RP/LT start() abort() hold() resume() create Status=Running OriginalJSDL=<JSDL..> ExecutionJSDL=<JSDL..> etc. Job1 Job3 Job2 Extends WS-RP/LT listDirectory(path) listFileProperties(path) createDirectory(path) changePermissions(path, perm) copy(path, path) rename(path, path) delete(path) import(path, protocol, isPipe) export(path, protocol, isPipe) create TS-Property AvailableFileSpace=127kb SupportedProtocols= {GridFTP, UPL, scp} etc. Work Home Root Temp create File Transfer 1 Sourcefile=c:/tmp/test.txt DestFile=/tmp/test.txt Transferred=300kb etc. Extends WS-RP/LT File Transfer 2 Atomic Service Interfaces based on WSRF Operations Service Resource Type Properties Job Management Service (JMS) Storage Management Service (SMS) File Transfer Service (FTS)

13. Hiding platform-specific information with Application Resources • Abstract job concept borrowed from UNICORE • No concrete platform specific information in job decription (paths, libraries, etc.) • Job will be incarnated on target system • For security and portability reasons! • Available applications can be queried from target system resource properties • Use JSDL Posix extensions to specify required application resources in submitted job

14. Transparency Virtualization of the computational resources requires the job definitions to be abstract. Non-abstract definition: /usr/bin/povray /var/uspaces/ec82a0f0-4b7d-bf2585/povray.pov Abstract definition: <Job> <Application name=“POVRay”/> <Field name=“Input”>povray.pov</Field> </Job>

15. Abstract Job Definitions Advantages of abstract job definitions: • Easy to use with brokers • Don’t need to disclose private information about target system configuration • One step job submission

16. File Transfers • Atomic Services support different protocols • FTP, GridFTP, plain HTTP(S), SOAP with Attachements, any other • Storage management announces available protocols via its resource properties • Client queries available protocols and selects appropriate one • GridFTP for large high-performance transfers • HTTPS or SOAP w/a to work with firewall limitations

17. Target System Target System Advanced Services: Workflow Execution Workflow Execution Service (TSS) • First implementation combining BPEL, JSDL and WSRF • Use Grid-specific BPEL subset • Allows integration into larger business processes • Implemented as WSRF service itself • Information about workflow (state, input/output files, etc.) is available through resource properties orchestrate BPEL-enabled JSDL Client Custom Web-service

18. Target System Target System Target System Advanced Services: Target System Registry Target System Registry • Registry keeps track of static and dynamic information • Hardware properties, available software, ... • Workload, available disk space, ... • TSS contacts registry on startup and when properties change (WS-Notification) • Clients and services query informations about target systems from registry Client Register/update Query Resource Broker

19. GridBeans Internals GridBean GPE Workflow Engine: • Contact arbitrary web-services • Perform XML evaluations Job Definition Atomic Target System: • Incarnate user’s task • Perform computations

20. GridBeans Financial Simulation Interface Visit POVRay Compiler Gaussian GridBean API GPE Client Framework Higher-level application services Web Portals Financial Simulations GPE Application Client Engineering Applications etc. GPE Expert Client GPE Client API Advanced services Atomic Services UNICORE/GS Workflow Resource Broker Globus Toolkit China Grid Support Package Target System Registry Grid Programming API • Easy to implement • Portable • Easy to maintain • Higher-level custom services take advantage of basic functionality provided by atomic and advanced services • Optional deployment of user interfaces through GridBeans

21. Setting up a GPE Grid

22. Adding a target system to a Grid • Start Target System Interface • Define Static Properties and Incarnation Rules • Start Service Container Configuration file describing hardware and software properties Target System Static Properties Lightweight Java component to perform the actual work Target System Interface (TSI) Configuration file defining systems specific paths to commands, libraries, etc. Incarnation Database (IDB) Service container Atomic services

23. Target System Target System Target System Target System Target System TSI TSI Service container TSI TSI TSI Atomic services Service container Atomic services Web Server Site A Service container Site B Atomic services Web Server Service container Advanced services Application services Site C Clients A GPE Inter-Grid

24. Globus Interface

25. Why Globus? • De-facto standard, widely used • Powerful, fast evolving • Has enough features to host GPE services

26. Differences between Globus and UNICORE/GS hosting environments de-facto standard

27. Portability GPE client core Globus client API implementation Unicore/GS client API implementation Globus atomic services interface UNICORE/GS GPE service core

28. Incarnations Database GPE uses incarnations database to store the templates of user’s jobs to incarnate. povray +I<SOURCE> +O<TARGET> +F<OUTPUT_FORMAT> +W<WIDTH> … fields Fields are taken from the abstract job definition and are merged with the default values for user or application. The system is very flexibly configurable.

29. Current State and Next Steps

30. GPE & GPE4GTK • GPE Alpha version has been released at UNICORE SourceForge project on June 1, 2005 • Full working version for demos and developers • GPE-enhanced GTK4 implementation • SourceForge project „GPE4GTK“ • First release on October 28, 2005 • Beta release on December 31, 2005 • UNICORE/GS support in European UniGridS project • Cooperation with China Grid to integrate GPE until end of 2006 • Possible cooperation with SimDAT

31. Summary Intel GPE... ...provides easy portable solution for different Grid infrastructures including UNICORE/GS and GTK4 ...provides a client framework to give users access to the infrastructure ...provides the GridBean concept and a programming API for Grid developers ...is available as Open Source today

32. Thank you !