1 / 22

AMGA Metadata Services: examples and usage scenarios

AMGA Metadata Services: examples and usage scenarios. Tony Calanducci INFN Catania Itacuruca, 04-15 December 2006. Outline. AMGA Basics Metadata concepts and AMGA terminology Collection and Schema management Queries Command Line Tools and API

beau
Download Presentation

AMGA Metadata Services: examples and usage scenarios

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AMGA Metadata Services: examples and usage scenarios Tony Calanducci INFN Catania Itacuruca, 04-15 December 2006

  2. Outline • AMGA Basics • Metadata concepts and AMGA terminology • Collection and Schema management • Queries • Command Line Tools and API • Use AMGA to exchange data among running grid jobs • Secure access to metadata EGRIS-1

  3. Metadata concepts and terminology • Entries – Entities/Objects which we are attaching metadata to • Metadata - List of attributes (including their values) associated with entries • Schema – A set of attributes • Attribute – key/value pair • Type – The type (int, float, string,…) • Name/Key – The name of the attribute • Value - Value of an entry's attribute • Collection – A set of entries associated with a schema (AMGA Collection are hierarchical organized) • Analogy to the RDBMS world: think of collections as tables, attributes as columns, entries as rows EGRIS-1

  4. Example: Movie Trailers • Movie trailers files saved on Grid Storage Elements and registered into a LFC File Catalogue • We have a LFN (Logical File Name) per movie files • We want to add metadata to describe movie content. Possible schema: • Title -- varchar • Rutime -- int • Cast -- varchar • LFN -- varchar • AMGA will be the repository of the movies’ metadata EGRIS-1

  5. gMOD: grid Movie On Demand • gMOD provides a Video-On-Demand service • User chooses among a list of video and the chosen one is streamed in real time to the video client of the user’s workstation • For each movie a lot of details (Title, Runtime, Country, Release Date, Genre, Director, Case, Plot Outline) are stored and users can search a particular movie querying on one or more attributes • Two kind of users can interact with gMOD: TrailersManagers that can administer the db of movies (uploading new ones and attaching metadata to them); GILDA VO users (guest) can browse, search and choose a movie to be streamed. EGRIS-1

  6. gMOD under the hood • Built on top of gLite services: • Storage Elements, sited in different place, physically contain the movie files • LFC, the File Catalogue, keeps track in which Storage Element a particular movie is located • AMGA is the repository of the detailed information for each movie, and makes possible queries on them • The Virtual Organization Membership Service (VOMS) is used to assign the right role to the different users • The Workload Management System(WMS) is responsible to retrieve the chosen movie from the right Storage Element and stream it over the network down to the user’s desktop or laptop EGRIS-1

  7. CE WN WN WN LFC File Catalogue Metadata Catalogue gMOD interactions VOMS Storage Elements Genius Portal AMGA get Role User Workload Management System EGRIS-1

  8. gMOD screenshot gMOD is accesible through the Genius Portal (https://glite-tutor.ct.infn.it) EGRIS-1

  9. Accessing AMGA services from UI/WNs • Command Line Tools: • mdclient (interactive, like mysql & psql CLI) • mdcli (run one single command, useful into scripts) • APIs: • C++ • Java • Python/Perl/PHP • Client configuration file: • /opt/glite/etc/mdclient.config (default for all UI users) • $HOME/.mdclient.config (overrides the default one) • My suggestion, copy the default into $HOME/.mdclient.config and customize it • Useful options to set up: • Host = egris196.eela.ufrj.br • Login = NULL (or your AMGA username we’ve created: itacurucaXX) • UseSSL = require • AuthenticateWithCertificate = 1 • UseGridProxy = 1 EGRIS-1

  10. Accessing AMGA services from UI/WNs • Be sure you have a valid proxy • with VOMS extensions if you set up Login = NULL (authenticated as the gilda generic user….all VO users will see your entries) • if you set up Login = yourusername the VOMS extensions will not take into account (so do not care about them) • Start the AMGA client with • $ mdclient • Try • whoami • pwd • help • grp_member EGRIS-1

  11. AMGA Basic Commands(I) • Entries/Collection listing • dir • ls (ls -l) • Entries/Collection permissions • acl_show entry | collection • stat entry | collection • Collection and schema creation : • createdir collection_pathname • createdir /egris/yoursurname • cd collection_pathname) --- change the current collection • cd /egris/yoursurname • createdir trailers • cd trailers • addattr dir attr_name attr_type • addattr . Title varchar • addattr . Runtime int • addattr . Cast varchar • addattr . LFC varchar EGRIS-1

  12. AMGA Basic Commands (II) • Inspecting collection’s schema: • listrattr fullpath_collectioname • (listrattr /egris/yoursurname/trailers • Sequence managements: • they are used to create autoincremental IDs • sequence_create seq_name dir [increment] [start value] • sequence_create seq1 /egris/yoursurname/trailers • sequence_next seq_name • sequence_next /egris/yoursurname/trailers/seq1 • Entries creations: • addentries entry_name (attribute_name value)+ • cd /egris/yoursurname/trailers • addentry 1 Title 'Matrix' Runtime 110 Cast 'Keanu Reeves‘ LFN 'lfn:/grid/gilda/tony/matrix.mov‘ • addentry 2 Title 'Notting Hill' Runtime 99 Cast 'Julia Roberts, Hugh Grant' LFN 'lfn:/grid/gilda/tony/notting.mov‘ • addentry 3 Title 'Anger Management' Runtime 120 Cast 'Adam Sandlan, Jack Nicholson' LFN 'lfn:/grid/gilda/scardaci/anger.mov' EGRIS-1

  13. Other basic useful commands • Set/edit attributes values: • setattr entry (attribute value)+ • Get attributes values: • getattr pattern (attribute)+ • getattr /egris/yoursurname/trailers/ Title LFN • Removing attributes • removeattr dir attr_name • Removing entries • rm pattern EGRIS-1

  14. Making queries • AMGA provides a SQL-like selectattr statement: • selectattr attrs... condition • where: • attrs is a space-separated list of attributes with the following format: collection_name:attribute_name (ex.: /egris/yoursurname/trailers:Title) • condition rapresents one or more condition to select entries. Allowed operators are comparison operator (<,>,=), logical operator (and, or), various operator (like, limit), etc..There is support also for aggregator operator (count) and order clause. • ex: • selectattr /egris/yoursurname/trailers:Title .:LFN 'like(Title, "Anger%")' • selectattr /egris/yoursurname/trailers:Title .:LFN .:Cast 'like(Cast, "%ndle%") and Runtime > 80' EGRIS-1

  15. Updating attributes on condition • selectattr and setattr can be used to update entries attributes’ values • updateattr allows to do multiple updating on condition and ATOMICALLY • updateattr pattern attr expression [attr expression]... condition • ex: • getattr /egris/yoursurname/trailers/ Runtime >> 2 >> 99 >> 1 >> 111 >> 3 >> 121 • updateattr /egris/yoursurname/trailers/ Runtime Runtime+1 'Runtime > 100‘ • getattr /egris/yoursurname/trailers/ Runtime >> 2 >> 99 >> 1 >> 112 >> 3 >> 122 EGRIS-1

  16. Access AMGA from WNs with mdcli • All commands available through mdclient can be issued with mdcli <command>. This will: • connect to the AMGA server defined into mdclient.config • run the command • exit to the shell • Very useful in shell script (ex. the one running in the WN during job execution) • Ex: • ID=`mdcli sequence_next /egris/yoursurname/seq1` • mdcli addentry /egris/yoursurname/${ID} LFN \'${LFN}\' Analyzed 0 • LFN=`mdcli selectattr $AMGA_HOME:LFN \'$AMGA_HOME:FILE = ${ID}\'` • N.B.: If you need to use single quote, escape them with \ EGRIS-1

  17. Use AMGA to exchange data among running jobs • Suppose we have two sets of jobs: • Producers: they generate a file, store on a SE, register it onto the LFC File Catalogue assigning a LFN • Consumers: they will take a LFN, download the file and elaborate it • AMGA can be used to share the information generated by the Producers, it could act as a “bag-of-LFNs” (bag-of-task model) from which Consumers can fetch file for further elaboration EGRIS-1

  18. Producers-Consumers example • Set up of the collection containing the “tasks” • createdir /egris/youraccount/tasks • cd /egris/youraccount/tasks • addattr . LFN varchar(255) • addattr . Analyzed int • addattr . MagicNumber int • listattr . >> LFN >> varchar(255) >> Analyzed >> int >> MagicNumber >> int • sequence_create seq1 /egris/youraccount/tasks EGRIS-1

  19. Producers-Consumers example • Producers script excerpts: • #Produce your data and save it on the SE and reg on LFC • LFN=‘lfn:/grid/gilda/tony/bomdia.txt’ • ID=`mdcli sequence_next /egris/yourname/tasks/seq1` • mdcli addentry /egris/yourname/tasks/${ID} LFN \'${LFN}\' Analyzed 0 • Consumers script excerpts: • AMGA_HOME=/egris/yoursurname/tasks • ID=`mdcli selectattr$AMGA_HOME:FILE \'$AMGA_HOME:Analyzed = 0\' | head -1` • if [ $ID ]; then; echo “Task to be processed found“; else; exit 1; fi • MAGIC=`echo $RANDOM` • mdcli updateattr $AMGA_HOME/$ID Analyzed 1 MagicNumber $MAGIC \'Analyzed = 0\' • MAGIC2=`mdcli selectattr $AMGA_HOME:MagicNumber \'$AMGA_HOME:FILE = ${ID}\'` • if [ $MAGIC=$MAGIC2 ]; then LFN=`mdcli selectattr $AMGA_HOME:LFN \'$AMGA_HOME:FILE = ${ID}\'` • else; echo "Entry already processed by someone else“; exit 1; fi • lcg-cp -v --vo gilda $LFN file:$PWD/temp.dat • #Consume your data EGRIS-1

  20. Demo • Thanks to Fabio for the creations of the scripts!!! EGRIS-1

  21. Secure your metadata • AMGA allows to restrict access to a collection (or even to specific entries) using ACLs. This can be done thanks to the commands provided to manage groups of users. • Here there are some useful commands: • whoami -- Print the name of the current user • user_listcred user -- Lists all possible credentials of a user. Only root can inspect other's credentials. • grp_member -- Lists the groups a user belongs to • acl_show colletionpath -- Shows all access controls for a directory • grp_create groupname -- Creates a new group. The group created will have the following name owner:groupName • grp_adduser groupname user -- Adds a user to a group • grp_show groupname -- Lists the members of a group • acl_add directory group rights -- Adds a new access control to a directory granting a group certain rights • acl_remove directory group -- Removes an access control for a group from a directory EGRIS-1

  22. References • AMGA Project Homepage • http://amga.web.cern.ch/amga/ • AMGA 1.2.7 User Manual • http://amga.web.cern.ch/amga/downloads/amga-manual_1_2_7.pdf • Exercise documentation from ISSGC’06: • http://www.dma.unina.it/~murli/ISSGC06/glite/public_html/amga.html • http://www.dma.unina.it/~murli/ISSGC06/glite/public_html/summaryExercise.htm • AMGA GILDA Wiki pages: • https://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAHandsOn • https://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAAdv EGRIS-1

More Related