1 / 27

JRA1 – Application and Infrastructure Grid Services

JRA1 – Application and Infrastructure Grid Services. Francisco Brasileiro Universidade Federal de Campina Grande – UFCG (Brazil) Diego Scardaci Istituto Nazionale di Fisica Nucleare – INFN (Italy) EELA-2 1 st Conference: All Members Meeting Bogotá, 24.02.2009. Agenda Role of the JRA1

darin
Download Presentation

JRA1 – Application and Infrastructure Grid Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. JRA1 – Application and Infrastructure Grid Services Francisco Brasileiro Universidade Federal de Campina Grande – UFCG (Brazil) Diego Scardaci Istituto Nazionale di Fisica Nucleare – INFN (Italy) EELA-2 1st Conference: All Members Meeting Bogotá, 24.02.2009

  2. Agenda • Role of the JRA1 • Services developed in the context of the activity • How to get help Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 2

  3. JRA1 Goals • Help in fostering the sustainability of the e-Infrastructure • Making the e-Infrastructure more interesting and wide spread by increasing its reach and its usability Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 3

  4. Increase the reach of the e-infrastructure by: • Allowing the scavenging of idle resources • Create the necessary mechanisms to allow resource centres that run the OurGrid middleware to co-exist with resource centres running gLite within the EELA platform • Provide some level of interoperation between these different kinds of resource centres and their associated applications • Allowing the execution of the grid middleware on top of platforms other than Scientific Linux • Port the gLite middleware to the Windows platform • Leveraging on the multi-platformcharacteristics of OurGrid Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 4

  5. Increase the usability of the e-infrastructure by: • Developing new application-oriented grid services • Leveraging the grid services provided by the OurGrid middleware to execute bag-of-tasks jobs • Facilitating the management of resource centres Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 5

  6. Services developed • Infrastructure-oriented • The main development here allows the co-existence of service grids (gLite-based) with opportunistic grids (OurGrid-based) • More details will be presented tomorrow • Application-oriented Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 6

  7. Grid2Win Porting of gLite middleware to Windows Platform Dario Russo, Fabio Scibilia, Elisa Ingrà INFN –Catania Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 7

  8. new users come in to the grid ! windows applications can run on the grid ! Grid2Win • The Grid2Win project aims to create a gLite User Interface (UI) and a gLite Computing Element (CE + WNs) running on Microsoft Windows. • Cygwin emulates a POSIX environment on Windows; • CE based on PBS and Microsoft Windows Server 2008 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 8

  9. Grid2Win – Main Features • We have ported: • Grid Security Infrastructure: Credentials management with VOMS extensions commands (voms-proxy-*) • Workload Management: Job submission and output retrieving commands (glite-wms-job-*) • File Transfer: Secure transfer with credential management GSIFTP (globus-url-copy) RFIO commands (rf*) • Data Management: File Catalogue Browsing commands (lfc-*) and data movement & replication (lcg-*) • Information System: Commandlcg-infosites and existing LDAP Explorer integrated in our GUI. Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 9

  10. Links • Grid2Win on EELA-2 forge: • https://forge.eu-eela.eu/projects/grid2win/ • Grid2Win on EELA-2 wiki: • https://grid.ct.infn.it/twiki/bin/view/EELA2/Grid2Win Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 10

  11. T-GSAF Transactional Grid Storage Access Framework Salvatore Scifo Consorzio Cometa – Catania Salvo Parisi IR&T Engineering s.r.l. - Catania Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 11

  12. GSAF as Design Help • We have common features, we have common problems we need a Design Pattern • Built on topof theGrid Metadata ServiceandGrid Data Service • collects and implements functionalities shared among applications according to “write once use anywhere” principle • reduces the knowledge gap • hiding the complexity and the fragmentation of the several gLite Data management APIs • exposing a unified interface more near to the developer mind (design patterns) rather than the Grid stuff details (API syntaxes) • acts as a black box providing • classes and related methods for applications located above • interfaces to extend the implemented capabilities Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 12

  13. GSAF - Data Grid Application solution • New • LFC API • CNS API • AMGA is supported • only one super interface for Data Grid Application • GSAF is an Object Oriented Framework • Java 100% - Design Pattern based • Clear design : object modeling / functional modeling • Solves the fragmentation of DMS APIs • Solves the natural (due to OGSA) inconsistence of services • Provides ACID Transactional Model (coming soon) • Road Map • RFIO java native implementation • SRM encapsulation • Grelc integration • Secure Storage integration • modular and extensible Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 13

  14. 1° Use Case: DMS Web Interface Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 14

  15. Transactional GSAF • Transaction Manager • Provides developers with a Transaction Pattern • autocommit(), execute(), commit(), rollback() • performs several data manipulation in Atomic Mode • Hold Runtime Exceptions to limit Data Inconsistence • System Crash Recovery to avoid Data Loss • Challenge • SOA of Grid middleware makes transactions impossible at server side • GSAF works at client side • Not full access to resources and services (locks) • Not cache mechanism to save temporary data (memory) • Not backup system to store previous data status (restore) • Not full access control to distributed resources (centralization) • ….. Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 15

  16. State of the Art • GSAF • Framework released • Adopters • ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts) • BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications • Aiuri project (COPPE/UFRJ - BRAZIL)‏:aims to implement a Grid Oriented platform to support data and text mining applications uses GS. • T-GSAF • Analysis and Design (Completed!) • Development, testing, production (Almost DONE!) Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 16

  17. Links • GSAF on EELA-2 forge: • https://forge.eu-eela.eu/projects/gsaf/ • GSAF on EELA-2 wiki: • https://grid.ct.infn.it/twiki/bin/view/EELA2/GSAF Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 17

  18. A Secure Storage Service for the gLite Middleware Diego Scardaci – INFN Catania Giordano Scuderi – UNICO S.R.L. Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 18

  19. Insider Abuse: the problem • A grid user could store sensitive data in a Storage Elements managed by external organizations. • Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format. • Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization). Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 19

  20. A Secure Storage service for the gLite Middleware • Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way. The service is composed by the following components: • Command Line Applications: commands integrated in the gLite User Interface to encrypt/upload and decrypt/ download files. • Application Program Interface: allows the developer to write programs able to manage confidential data . • Keystore: a new grid element used to store and retrieve the users’ keys. Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 20

  21. Links • Secure Storage on EELA-2 forge: • https://forge.eu-eela.eu/projects/securestorage/ • Secure Storage on EELA-2 wiki: • https://grid.ct.infn.it/twiki/bin/view/EELA2/SecureStorage Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 21

  22. WatchDogMonitoring and controlling job execution on the WorkerNode Riccardo Bruno INFN – Catania Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 22

  23. Why do we need a WatchDog ? • Why • Especially long term jobs require to be monitored and controlled during their execution. • How • Perform job control and monitoring using grid services in the less invasive way. • Observations • Almost all jobs submitted on the grid are piloted by shell scripts • Shell scripting allow to get precious info in case of faults • Shell scripting can pilot more complex batch execution • Both AMGA and SE+LFC can be used as the simplest IS on the grid. • lfc-* and lcg-* tools already available for file creation and retrieve • The latency of CLI tools for the storage is very low compared to long term jobs • Requirements: • Monitor job execution watching snapshot of files produced by the job execution • File snapshot will be reported on LFC+SE or AMGA servers • It would be useful to configure the monitoring tool accordingly to the user needs • Few shell environment variables can be used to configure the watchdog tool • Control the job execution accessing directly on the WN • It is possible to send commands on the WN Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 23

  24. How does the WatchDog work ? UI UI WN WN • The Watchdog is a shell script to be included in the JDL main script. • Some watchdog features: • It starts in background before to run the long term job • The watchdog runs as long as the main job • The main script can control, stop and wait until the watchdog has finished • Easily and highly configurable and customizable • The watchdog does not compromise the CPU power of the WN UI JDL WN Commands or Scripts File snapshots Amga LFC+SE Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 24

  25. Links • Watchdog on EELA-2 forge: • https://forge.eu-eela.eu/projects/watchdog/ • WatchDog on EELA-2 wiki: • https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 25

  26. Other JRA1 Services • GENIUS: - https://glite-tutor.ct.infn.itGFAL Java API: - https://grid.ct.infn.it/twiki/bin/view/EELA2/GFALJAVAAPI - https://grid.ct.infn.it/twiki/GFAL/gLibrary/DRI: - https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-UF3.pdf - https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-CBMS08.pdfStorage Accounting (SAGE): - https://forge.eu-eela.eu/projects/sage/ All information available on the JRA1 section of the EELA-2 wiki: http://wiki.eu-eela.eu → JRA1 Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 26

  27. JRA1 Resources • JRA1 Section in EELA-2 web site • Contains a summary of the activity goals and points to the Wiki site • https://www.eu-eela.eu/ • JRA1 Wiki Site • Contains descriptions and links for each JRA1 services • https://grid.ct.infn.it/twiki/bin/view/EELA2/JRA1Services • EELA-Forge Site • All released software • https://forge.eu-eela.eu/ • Contact us at: • Francisco Brasileiro (fubica@dsc.ufcg.edu.br) • Diego Scardaci (diego.scardaci@ct.infn.it) Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 27

More Related