1 / 36

iRODS h ands- o n tutorial

iRODS h ands- o n tutorial. 8 th May 2014. Christine Staiger SURFsara. Amsterdam, 8 th May 2014. Contents. The Basics Installation Tools for administration Adding data and metadata Complex storage systems Federations between data grids Resources and Compound resources. The Basics.

silas
Download Presentation

iRODS h ands- o n tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. iRODShands-on tutorial 8th May 2014 Christine Staiger SURFsara Amsterdam, 8th May 2014

  2. Contents The Basics Installation Tools for administration Adding data and metadata Complex storage systems Federations between data grids Resources and Compound resources

  3. The Basics

  4. Demo Data Grids 10 Virtual machines:sshuser@IP 4

  5. Installation • Execute ./irodssetup in the folder iRODS • Export the icommands • export PATH=$PATH:/home/user/iRODS/clients/icommands/bin • ~/iRODS/config/irods.config # Database configuration $DATABASE_TYPE = 'postgres'; … $DATABASE_HOST = 'localhost'; $DATABASE_PORT = '5432'; $DATABASE_ADMIN_PASSWORD = 'irods2'; $DATABASE_ADMIN_NAME = ’alice'; # iRODS configuration IRODS_HOME = '/home/alice/iRODS'; $IRODS_PORT = '1247’; … $IRODS_ADMIN_PASSWORD = ’alice'; $IRODS_ICAT_HOST = ''; 5

  6. iRODS i-commands Documentation: https://wiki.irods.org/index.php/icommands (wedon’tprovide a exhaustivelist of i-commands) 6

  7. iRODS Control Execute ./irodsctlargin the folder iRODS Restarting, starting and stopping the iRODS and iCATserver with arg = restart, start, stop Restarting, starting and stopping only the iRODSserver with arg = irestart, istart, istop Initialise an account and log in with iinit

  8. .irodsEnv file # iRODS personal configuration file. # # This file wasautomaticallycreatedduringiRODSinstallation. # CreatedMon Jun 18 13:13:27 2012 # # iRODS server host name: irodsHost’surfsara-cloud' # iRODS server portnumber: irodsPort1247 # Default storageresourcename: irodsDefResource'demoResc' # Home directory in iRODS: irodsHome'/alicesZone/home/alice' # Currentdirectory in iRODS: irodsCwd'/alicesZone/home/alice' # Accountname: irodsUserName’alice' # Zone: irodsZone‘alicesZone' The .irodsEnv determines the zone and the user for the icommands-client. You can only have one such file at a time. 8

  9. Theicommandsclient • icommands are automaticallyinstalledwhenyouinstalltheiRODS server • UserswillhavetoinstallthemseperatelytoconnecttoaniRODS server • Categories: • Informational • UNIX and FTP like • Metadata • Functional 9

  10. ihelp • ihelp • Lists all commands • ihelp -h • Usage: ihelp [-ah] [icommand] Display i-commands synopsis or a particular i-command help text • Options are: • -h this help • -a print the help text for all the i-commands • Run with no options to display a synopsis of the i-commands • iRODS Version 3.3.1 February 2014 ihelp 10

  11. ienv • ListstheiRODSenvironment variables. • Equivalentto “iinit -l” • Usage: ienv [-h] NOTICE: Release Version = rods3.3.1, API Version = d NOTICE: irodsHost=surfsara-cloud NOTICE: irodsPort=1247 NOTICE: irodsDefResource=demoResc NOTICE: irodsHome=/alicesZone/home/alice NOTICE: irodsCwd=/alicesZone/home/alice NOTICE: irodsUserName=alice NOTICE: irodsZone=alicesZone 11

  12. iuserinfo • Show informationaboutyouriRODSuseraccountortheentereduser • Usage: iuserinfo [-vVh] [user] name: alice id: 10007 type: rodsadmin zone: alicesZone info: comment: create time: 01396887567: 2014-04-07.18:19:27 modify time: 01396887567: 2014-04-07.18:19:27 member of group: alice member of group: public 12

  13. imiscsvrinfo • Connecttothe server and retrievesomebasic server information • Can be used as a simple test forconnectingtothe server. • Usage: imiscsrvinfo [-hvV] RCAT_ENABLED relVersion=rods3.3.1 apiVersion=d rodsZone=alicesZone up 0 days, 5:54 13

  14. ilsresc • ilsresclistsiRODSresources and resource-groups • Usage: ilsresc [-lvVhA] [Name] alice@sara-cloud:~$ ilsresc demoResc alice@sara-cloud:~$ ilsresc –l demoResc resource name: demoResc resc id: 10010 zone: alicesZone type: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault … 14

  15. Somecommands • Unix-like: • ils, ipwd, icd, • imkdir, irm, icp • ichmod, ipasswd, • irsync, ichksum, imv • FTP-like: • iinit, iput, iget, iexit • Test: Create folder in thelogicalnamespace • ipwd • imkdirNewFolder • ilsNewFolder • irm–r NewFolder 15

  16. icommands - ACLs ListACLs (Access Control Lists) in a collection: alice@sara-cloud:~$ ils -A /alice/home/alice: ACL - alice#alicesZone:own Inheritance - Disabled 16

  17. Administration • Change to admin mode: iadmin • Quit admin mode:q • Make/delete a user: • mkuser, rmuser, moduser • (iadmin) mkuser user1 rodsuser • (iadmin) moduser user1 password ***** • Parameters and their values: iadminlt, iadminltuser_type • iadminmkresc, iadminrmresc • Create new resource: • mkdir Data (in your home, not in the iRODS environment) • iadminmkrescNewDataResc “unix file system” cache • 145.100.57.139 “/home/alice/Data” 17

  18. Put and replicate data • Add the file /home/login/put1.txt to resource • iput –v put1.txt • iput -h • ichksum put1.txt (check sum of copy in Vault) • md5sum put1.txt (check sum of file in /home/login) • iput -RNewDataResc put2.txt • Replicate data: irepl -RNewDataResc put1.txt • Trim number of copies: itrim -SdemoResc-N 1 put1.txt 18

  19. Access Control Lists • Changepermissionfor a user: • ichmodread user1 put1.txt • List access: ils -A • Changepermissionsforgroup “public”: • Make a directory in theiRODSenvironment: imkdirtemp • Makedirectoryreadablefor “public”:ichmod -r readpublictemp • ils –A • /bobsZone/home/bob/temp: • ACL - bob#bobsZone:owng:public#bobsZone:readobject • Inheritance -Disabled 19

  20. iRODS Metadata • Commands: imeta, iquest, idbo • Addmetadata: • imetaadd -d put1.txt “Date” “08.10.2013” • imetaadd-dput1.txt “AnotherDate” “04.04.2014” • imetals -d put1.txt • Removemetadata: • imetarm -d put1.txt “Date” “08.10.2013” • QueryingtheiCAT: • iquest"SELECT DATA_NAME,DATA_CHECKSUM WHERE DATA_RESC_NAME like ’%Resc%’” • DATA_NAME = put1.txt • DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c 20

  21. Complex storage systems

  22. Federation between data grids • List all existing zones: ils / • Zone A acknowledges Zone B and vice-cersa • iadminmkzone B remote Host:Port • Zone A adds remote user from Zone B: • iadminmkuseruser#Brodsuser • Example: Alice and Bob: • Alice acknowledges Bob’s zone: • iadminmkzonebobsZoneremote 145.100.57.74:1247 • Bob acknowledges Alice’s zone: • iadminmkzonealicesZoneremote 145.100.57.66:1247 • Alice creates remote user Bob: • iadminmkuserbob#bobsZonerodsuser • Now Bob can list and alter files in Alice’s zone: ils /alicesZone 22

  23. Troubleshooting ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit iRODS/server/config/irodsHost localhostyou.rIP.XX.XXsurfsara-cloud localhost 12.0.0.1 ./irodsctl restart 23

  24. Data Replicationbetweengrids • Bob copies a file fromhiszonetohisaccount in Alice’szone • irsync –R demoResc i:/bobsZone/home/bob/put1.txt • i:/alicesZone/home/bob#bobsZone/put1.txt • -R specifiesthe target resource • Bob copies a frilefromhis home directorytoAlice’szone • irsync –R demoResc put3.txt • i:/alicesZone/home/bob#bobsZone/put3.txt • Bob copies a file fromAlice’szonetohis home directory • mkdir test • irsync i:/alicesZone/home/bob#bobsZone/put3.txt • test/put3.txt • Bob deletes a file in Alice’szone • irm /alicesZone/home/bob#bobsZone/put3.txt 24

  25. RemoteResources InstalliRODS, iCATdisabled BuildaniRODS server [yes]? Makethis Server ICAT-Enabled [yes]? no Host runningiCAT-enablediRODS server? 145.100.58.11 … Resourcename [demoResc2]? myResc2 Resourcestorageareadirectory [/home/irodsadmin/iRODS/Vault]? /home/irodsadmin/iRODS/alicesVault … TheirodsServerwillauthenticatetotheother servers via anexistingiRODSadministratoraccount. ExistingiRODSadminloginname [rods]? alice Password [rods]? ******** Enterthezone as defined in your ICAT iRODSzonename [tempZone]? alicesZone

  26. ResourceGroups • ExerciseResourceGroups • Createtwo data resources of type cache • Groupthem: iadminatrggroupNameresourceName • Deposit data in resourcegroup: iput file groupName • Addsome data totheresourcegroup • Checkwithilswherethe file islocatedvirtually and physically • Howtoautomaticallyreplicate data toresources in thesameresourcegroup?

  27. iRODSMicroservices Micro-services are small, well-defined procedures/functions that perform a simple task (https://www.irods.org/index.php/iRODS_Micro-services) Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, … To convert any C procedure (createCollection) into a microservice create an interface routine (msiCreateCollection) 27

  28. iRODSRules • Example: replicate.r • replicate{ • msiWriteRodsLog("startingreplication", *status); • msiWriteRodsLog("source = *source", *status); • msiDataObjRsync(*source, "IRODS_TO_IRODS", "null", *destination, *rsyncStatus); • } • INPUT *source="/bobsZone/home/bob/put3.txt", • *destination="/alicesZone/home/bob#bobsZone/put3.txt" • OUTPUT ruleExecOut • Execute irule -vF ~/rules/replicate.r • Check with ils/alicesZone/home/bob#bobsZone 28

  29. Time delayed and periodicjobs • Exampletest.r • myTestRule{ • delay("<PLUSET>60s</PLUSET>"){ • msiWriteRodsLog("HelloWorld.", *status); • } • } • INPUT *A="1" • OUTPUT ruleExecOut • Writesthe output toiRODS/server/log/reLog*, alsocheckrodsLog* • Use iqstat, iqdeland iqmodtolist, delete and modifydelayed rules. • delay(”<PLUSET>5m</PLUSET>EF>1h</EF>”) • Startjob in 5 minutes and restartitevery 1 hour • Predefined rules: iRODS/server/config/reConfigs/core.re 29

  30. Compoundresources List resource types: iadminltresc_type A compound resource consists of cache and a storage resource (s3, MSS universal driver) Storage resource can be located on several different severs Example: Universal Mass Storage User writes data to cache, system replicates data to storage and deletes it from cache admin mkrescCache "unix file system" cache 145.100.57.197 /home/alice/Cache iadminmkrescmssStore 'MSS universal driver' compound 145.100.57.197 /home/alice/MSS iadminatrgmssGroup Cache iadminatrgmssGroupmssStore

  31. Compoundresources The mssStore can be of any hardware type which might require certain protocols to put and get data, e.g. gridFTP The resource type 'MSS universal driver’ triggers iRODS to look for a file iRODS/server/bin/cmd/univMSSInterface.sh to overwrite the put and get functions Template in iRODS/server/bin/cmd/examples Example file in TODO

  32. The Universal Mass Storage Interface Logging: _log() LOGFILE=/home/irodsadmin/univMSSInterface.logChange the path Copy between cache and storage: syncToArch(), stageToCache() _log 2 syncToArch "entering syncToArch()=$*" _log 2 syncToArch "executing: /bin/cp -f \"$1\" \"$2\"" /bin/cp -f "$1" "$2” error=$? if [ $error != 0 ] # copy failure then STATUS="FAILURE" fi _log 2 syncToArch "The status is $error ($STATUS)" return $error

  33. Replicate data replicateDiskCache { delay("<PLUSET>30s</PLUSET><EF>15m</EF>") { … *Condition = "DATA_RESC_NAME = '*CacheRescName' AND COLL_NAME like '*Collection%'"; msiMakeGenQuery("DATA_NAME, COLL_NAME, DATA_SIZE, order(DATA_CREATE_TIME)",*Condition,*Query2); msiExecGenQuery(*Query2,*List); while ( *ContInxOld > 0 ) { foreach ( *List ) { msiGetValByKey(*List,"DATA_NAME",*D); msiGetValByKey(*List,"COLL_NAME",*C); msiDataObjRepl("*C/*D","backupRescName=*PnfsRescName++++ rescName=*CacheRescName++++irodsAdmin=",*status)} } }}

  34. Replicate data Save as YourFilename.re in iRODS/server/config/reConfigs/ Edit server/config/server.config reRuleSetcore,YourFilename irulereplicateDiskCache"*Collection=/yourZone%*PnfsRescName= yourMssStore%*CacheRescName=yourCache" ruleExecOut alice@surfsara-cloud:~/iRODS$iput -R mssGroup ../put1.txt alice@surfsara-cloud:~/iRODS$ ils -l /alicesZone/home/alice: alice 0 Cache 7 2014-04-24.09:26 & put1.txt Check with iqstat -l when the rule will be executed Some time later: alice@surfsara-cloud:~/iRODS$ ils -l /alicesZone/home/alice: alice 0 Cache 7 2014-04-24.09:26 & put1.txt alice 1 mssStore 7 2014-04-24.09:36 & put1.txt No space!

  35. Purge Cache Write a rule to purge the cache Purge cache when data is replicated and if not enough space left in cache Call by: irulepurgeDiskCache"*Collection=/testZone%*PnfsRescName=mssStore %*CacheRescName=Cache%*MaxSpAlwdTBstring=2" ruleExecOut MaxSpAlwdTBstring: the size when to purge, string

  36. Thank you !

More Related