1 / 49

EELA-2 JRA1 Services

EELA-2 JRA1 Services. Diego Scardaci INFN (Italy) Joint EELA-2/EGEE-III Tutorial for Trainers Catania (Italy), 01.07.2008. Outline. NA3/JRA1 Synergies EELA-2 Services already available : GENIUS GFAL Java API gLibrary/DRI Grid2Win Secure Storage Storage Accounting (SAGE)

tuari
Download Presentation

EELA-2 JRA1 Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EELA-2 JRA1 Services Diego Scardaci INFN (Italy) Joint EELA-2/EGEE-III Tutorial for Trainers Catania (Italy), 01.07.2008

  2. Outline NA3/JRA1 Synergies EELA-2 Services already available: GENIUS GFAL Java API gLibrary/DRI Grid2Win Secure Storage Storage Accounting (SAGE) Transactional Grid Storage Access Framework (T-GSAF) Watchdog All information available on the JRA1 section of the EELA-2 wiki: http://wiki.eu-eela.eu → JRA1 Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  3. NA3 & JRA1 Synergies Applications Support Provides support to EELA-2 Applications NA3 REQUIREMENTS IDENTIFICATION SUPPORT Grid Services Developing Develop Services that will decisively contribute to a durable, versatile and cost-effective use of the e-Infrastructure Applications Requirements Identify Services & Tools useful for EELA-2 Applications NA3/JRA1 JRA1 GRID SERVICES DEVELOPMENT Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  4. EELA-2 Services In the next slides we will see for each EELA-2 Service: • A brief introduction • Main features • Links to installation packages, web/wiki site, documentations, etc. • RoadMap All EELA-2 Services will soon be submitted under the evaluation of the EGEE RESPECT programme Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  5. GENIUS Gilda Team & Nice S.R.L. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  6. GENIUS GRID portal • It can be accessed from everywhere and by “everything” (desktop, laptop, PDA, cell phone). • The same user interface to several back-ends. • All available grid services incorporated in a logic way, just “one mouse click away”. • Layout easily understandable and user friendly. • Secure at all levels: • secure for web transactions • secure for user credentials • secure for user authentication • secure at VO level Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  7. Main advantages Thanks to the GENIUS and EnginFrame (developed by the Italian Company NICE srl) user can • Interact with files on the UI • Send jobs to the Grid • Manage the data belonging to the given Virtual Organization. Moreover: • No needs of a particular Operating System and middle-ware running on the client side Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  8. The Genius Home Page (https://glite-tutor.ct.infn.it) Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  9. GFAL Java API Diego Scardaci INFN – Catania Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  10. GFAL Java API Grid File Acces Library (GFAL): • Client POSIX (like) I/O library for directly data access. Aim: • Provide GFAL API to Java Developers. Why Java? • Reduce developing time; • More Java developers are “available”; • More applications are “pure Java” applications. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  11. GFAL Java API Classes • GFalFile: implements methods to manage remote files (stored on a SE). • GFalDirectory: implements methods to manage SE remote directory. • GFalUtilities: provides some important utility methods. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  12. References • GFAL C API: http://grid-deployment.web.cern.ch/grid-deployment/gis/GFAL/GFALindex.html • GFAL Java API wiki page: https://grid.ct.infn.it/twiki/bin/view/GILDA/APIGFAL • GFal Java API Javadoc: https://grid.ct.infn.it/twiki/GFAL/ Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  13. A GRID based platform to host multiple repositories for digital content A. Calanducci, J.M. González, R. Ramos, M. Rubio, D.TcaciEELA-II Kick-Off Meeting 20th-23rd April 2008 – Trujillo (Spain) Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  14. What is gLibrary/DRI ? • A platform to easily build digital repositories on a grid infrastructure • Digital Repository: annotated digitalized data offered in a structured manner to users • Features: • Host multiple repository of arbitrary structure (defined by repository providers) • Totally grid (gLite) based: • Authentication/Authorization based on GSI/VOMS • Digitalized data files saved on any SRM Storage Element • Annotations stored on Metadata Service (AMGA) or RDBMS • Specific algorithm to process data run through the grid WMS • Easy-to-use: AJAX web front-end offering a-la-iTunes browsing • a DRI APIs (Java based) available to easily deploy new repositories Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  15. gLibrary/DRI web interface Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  16. gLibrary/DRI API • One repository must provide • A description of its navigational structures (trees, filters) and viewers • A description of its data model • A storage engine (for data model persistence) • The DRI API specification describes HOW this is provided • In practise, three Java modules should implement the following interfaces: • DRIUIInterface for describing trees, filters and viewers • DRIStorageInterface for storing and retrieving data • DRINodeInterface for defining repository data model • One repository can • Make its own implementation of the specification • Use the default one provided Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  17. Mammograms rep with viewer Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  18. Technologies involved • Web interface • Servlets, php, java bridge • Applets • For user authentication with their VO certificate • For viewers implementation • Java Introspection • XML • AMGA, gLite Java APIs Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 18

  19. Current status • Engine deployed and working, API and default implementation (save metadata on AMGA) working • MGPlus repository implemented on DRI • Generic Uploader ready • Current work: • Interaction with LFC/SE improvements • Multi-node selection for loading and viewing • To Do: • Deployment system for new repositories • Generic viewer for repository that doesn’t provide a specific one • Run grid jobs with selected algorithms with data coming from repositories • Secure Storage/GSAF Integration • Service Orientation for the DRI Engine (Web Service) • Documentation, packaging, installation systems to be RESPECT compliant Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  20. References • gLibrary/DRI presentation at the EGEE User Forum III in Clermont-Ferrand • https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-UF3.pdf • Paper submitted to the 21st International Symposium on Computer Based Medical Systems (CBMS 2008) • https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-CBMS08.pdf • Chapter submitted for the “Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine and Healthcare” • https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-chapter.doc • Poster presented at the User Forum III • https://glibrary.ct.infn.it/glibrary/downloads/DRI/poster_glibraryDRI%20final.pdf • Developer’s web server: • https://dri-dev.ceta-ciemat.es/index.php • Wiki (internals only) • http://www.ceta-ciemat.es/projects/CETA-DRI/ Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  21. Grid2Win Porting of gLite middleware to Windows Platform Dario Russo, Fabio Scibilia, Elisa Ingrà INFN –Catania Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  22. The Grid2Win project aims to create a gLite User Interface (UI) and a gLite Computing Element (CE + WNs) running on Microsoft Windows. new users come in to the grid ! windows applications can run on the grid ! Grid2Win • Cygwin emulates a POSIX environment on Windows; • CE based on PBS and Microsoft CCS Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  23. Grid2Win – Main Features • We have ported: • Grid Security Infrastructure: Credentials management with VOMS extensions commands (voms-proxy-*) • Workload Management: Job submission and output retrieving commands (edg-job-*) • File Transfer: Secure transfer with credential management GSIFTP (globus-url-copy) RFIO commands (rf*) • Data Management: File Catalogue Browsing commands (lfc-*) and data movement & replication (lcg-*) • Information System: Commandlcg-infosites and existing LDAP Explorer integrated in our GUI. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  24. Links You can download the last versione of Grid2Win from the project homepage : http://grid2win.gilda-forge.ct.infn.it/new/index.php Installation and configuration instructions (for VO different than GILDA) can be found here: https://grid.ct.infn.it/twiki/bin/view/GILDA/Grid2WinGUI The online Documentation: http://grid2win.gilda-forge.ct.infn.it/onlineDocumentation/index.php You can read more about the project here: http://grid2win.gilda-forge.ct.infn.it/Grid2Win_ETNGrid2007_final.pdf Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  25. T-GSAF Transactional Grid Storage Access Framework Salvatore Scifo Consorzio Cometa – Catania Salvo Parisi IR&T Engineering s.r.l. - Catania Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  26. gLite C/C++ API weakness • LGC Utils is not a framework • LGC Utils is not GOOD for Data Grid Application • LGC Utils is the only super interface • Doesn’t include GsiFTP • Doesn’t include GFAL • Misses Metadata integration (AMGA) • messyoverlap among different semantic APIs • GFAL is only a POSIX-like file manage interface, it should not wrap LFC API than just RFIO would be enough Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  27. GSAF -Data Grid Application solution • New • LFC API • CNS API • AMGA is supported • only one super interface for Data Grid Application • GSAF is an Object Oriented Framework • Java 100% - Design Pattern based • Clear design : object modeling / functional modeling • Solves the fragmentation of DMS APIs • Solves the natural (due to OGSA) inconsistence of services • Provides ACID Transactional Model (coming soon) • Road Map • RFIO java native implementation • SRM encapsulation • Grelc integration • Secure Storage integration • modular and extensible Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  28. GSAF as Design Help • We have common features, we have common problems we need a Design Pattern • Built on topof theGrid Metadata ServiceandGrid Data Service • collects and implements functionalities shared among applications according to “write once use anywhere” principle • reduces the knowledge gap • hiding the complexity and the fragmentation of the several underlying APIs • exposing a unified interface more near to the developer mind (design patterns) rather than the Grid stuff details (API syntaxes) • acts as a black box providing • classes and related methods for applications located above • interfaces to extend the implemented capabilities Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  29. 1° Use Case: DMS Web Interface Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  30. Transactional GSAF • Transaction Manager • Provides developers a Transaction Pattern • autocommit(), execute(), commit(), rollback() • performs several data manipulation in Atomic Mode • Hold Runtime Exceptions to limit Data Inconsistence • System Crash Recovery to avoid Data Loss • Challenge • SOA of Grid middleware makes transactions impossible at server side • GSAF works at client side • Not full access to resources and services (locks) • Not cache mechanism to save temporary data (memory) • Not backup system to store previous data status (restore) • Not full access control to distributed resources (centralization) • ….. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  31. State of the Art • GSAF • Framework (v1.01) released • Candidate at RESPECT program • Adopters • ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts) • BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications • Aiuri project (COPPE/UFRJ - BRAZIL)‏:aims to implement a Grid Oriented platform to support data and text mining applications uses GS. • References • http://gilda-forge.ct.infn.it/projects/gsaf/ • https://grid.ct.infn.it/twiki/bin/view/PI2S2/GSAF • http://www.ieeexplore.ieee.org/xpl/freeabs_all.jsp?isnumber=4407105&arnumber=4407174&count=98&index=68 • T-GSAF • Analysis and Design (progress) • Development, testing, production (roadmap) Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  32. A Secure Storage Service for the gLite Middleware Diego Scardaci – INFN Catania Giordano Scuderi – UNICO S.R.L. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  33. Insider Abuse: the problem • A grid user could store sensitive data in a Storage Elements managed by external organizations. • Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format. • Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization). Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  34. Insider Abuse: the solution SECURE ENVIRONMENT USER (VIRTUAL) ORGANIZATION SE File Encryption /Decryption Encrypted File Key Encrypted File SE Key Repository Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  35. A Secure Storage service for the gLite Middleware • Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way. The service is composed by the following components: • Command Line Applications: commands integrated in the gLite User Interface to encrypt/upload and decrypt/ download files. • Application Program Interface: allows the developer to write programs able to manage confidential data . • Keystore: a new grid element used to store and retrieve the users’ keys. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  36. lcg-scr: Encryption and Storage Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  37. lcg-scp: Retrieval and Decryption Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  38. The Keystore • The Keystore is a new grid element used to store and retrieve the users’ key in a secure way. • is identified by an host X.509 digital certificate; • all its Gridtransactions are mutually authenticated and encrypted as required by the GSI model; • should be placed in a trusted domain and should be appropriately protected by undesired connections; • is a black box with a single interface towards the external world. This interface accepts only GSI authenticated connections; • the client request is processed only if the client is a member of a enabled users list and/or it belongs to an enabled Virtual Organization; • if the client want to retrieve a key, the keystore checks if the request is coming from an authorized user inserted on the ACL associated to the request key. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  39. Storage Accounting for Grid Environments Fabio Scibilia Consorzio COMETA Catania, Italy Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  40. SAGE: Accounting of Storage Resources • Accounting in gLite: • CPU: DGAS makes accounting of CPU cycles consumed by jobs. • Storage: Currently no system exists. • SAGE: Storage Accounting for Grid Environments • Measures disk space used by files through a scandisk and by intercepting accesses to files • Will be integrated with the database of DGAS (HLR). • Allow users to create reports on the usage of disk space • Measure of unit: • Considers both file life time and its space in bytes • Consumption is defined as the integral of the file size (bytes) in time (seconds). • http://sage.gilda-forge.ct.infn.it/ Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  41. Features • Interacts with the Mass Storage System (MSS) to resolve local files into grid files. • The local MSS (DPM by now) is unaware of being monitored • Can be extended to more with more MSS by adding some simple C++ classes to the library • It can scale to many disk servers in the same pool • It uses MySql as DBMS to store accounting data • Accounting is made per file, per user, per VO, per day. Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  42. Example of web reporting in SAGE Disk usage in MB*hours All the whole site Detailing on these VO Billing of users (one by one) VO admin report generation  Consider this storage  Consider these VO (P.S.: you must be a VO-admin to ask for this) In this time range  Tell me user subjects Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  43. Links SAGE on GILDA forge - http://sage.gilda-forge.ct.infn.it/ SAGE on EELA-2 Wiki: -https://grid.ct.infn.it/twiki/bin/view/EELA2/StorageAccounting Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  44. WatchDogMonitoring and controlling job execution on the WorkerNode Riccardo Bruno INFN – Catania Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  45. Why do we need a WatchDog ? • Why • Expecially long term jobs require monitored and controlled during their execution. • How • Perform job control and monitoring using grid services in the less invasive way. • Observations • Almost all jobs submitted on the grid are piloted by shell scripts • Shell scripting allow to get precious info in case of faults • Shell scripting can pilot more complex batch execution • Both AMGA and SE+LFC can be used as the simplest IS on the grid. • lfc-* and lcg-* tools already available for file creation and retrieve • The latency of CLI tools for the storage is very low compared to long term jobs • Requirements: • Monitor job execution watching snapshot of files produced by the job execution • File snapshot will be reported on LFC+SE or AMGA servers • It would be useful to configure the monitoring tool accordingly to the user needs • Few shell environment variables can be used to configure the watchdog tool • Control the job execution accessing directly on the WN • It is possible to send commands on the WN Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  46. How does the WatchDog work ? UI UI WN WN • The Watchdog is a shell script to be included in the JDL main script. • Some watchdog features: • It starts in background before to run the long term job • The watchdog runs as long as the main job • The main script can control, stop and wait until the watchdog has finished • Easily and highly configurable and customizable • The watchdog does not compromise the CPU power of the WN UI JDL WN Commands or Scripts File snapshots Amga LFC+SE Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  47. Links • WatchDog wiki page • https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  48. EELA-2 Apps. & JRA1 services(a proposal) (1/2) Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

  49. EELA-2 Apps. & JRA1 services(a proposal) (2/2) Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008

More Related