iRODS 4.0 Tutorial I. Getting Started

1. iRODS 4.0 TutorialI. Getting Started Leesa Brieger

2. iRODS Tutorial Preview • iRODS Getting Started • Unix client (icommands) • Usage • iRODS Data Grid Administration • Installing server and iCAT • Changing default settings • Setting up users • Installing resource server and adding new resources to a data grid/zone • Federating with other grids/zones, remote users • Microservices and rules for policy implementation and enforcement

3. iRODS Getting Started

4. iRODS Info • Main page: http://irods.org • Github repositories: https://github.com/irods/ (Request a guest account to get access.) • Github tickets (issues): https://github.com/irods/irods/issues?state=open • Chat list: irods-chat@irods.org • The original iRODS wiki: https://wiki.irods.org • The iRODS Manual: http://irods.org/dev/wp-content/uploads/2014/03/irods-manual-4.0.0.pdf

5. iRODS Books Available from Amazon • The integrated Rule-Oriented Data System (iRODS) Micro-service Workbook http://www.amazon.com/dp/1466469129 • An updated microservices reference book is coming out soon.

6. iRODS Download • Download link for binary packages from the iRODS main page http://irods.org/download/ • Open source, BSD license • Binary packages – rpmand deb – available for CentOS, Red Hat, Debian, Ubuntu, SUSE. Planned: Mac OSX, Windows • See http://irods.org/2014/03/irods-4-0-0-released/ • 4.0 tested platforms: CentOS 5 & 6, SuSE 11 & 12, Ubuntu 10 & 12 • iRODS source is available via GitHub https://github.com/irods/irods • Other iRODS packages and plug-ins are available via GitHub at https://github.com/irods

7. Install iRODS Unix Client (icommands) • Download the client package for your platform at https://irods.org/download/ • Install the package • Ubuntu: sudo dpkg –i irods-icommands-4.0.0-64bit.deb • CentOS 6: sudo rpm –i irods-icommands-4.0.0-64bit-centos6.rpm • This will: • install the icommands to your path • advise you to create an environment file: ~/.irods/.irodsEnv

8. The iRODS Environment File • ~/.irods/.irodsEnv • The .irodsEnv file determines which data grid (zone) the icommands client connects to- it determines your whole iRODS environment • server host • iRODS user name • port number • default resource • home directory • working directory • … • Use multiple environment files to choose from among many grids • only one file at a time has the name .irodsEnv • such environment changes are only feasible for Unix accounts that are not the “irods” service account

9. Working with a Demo Data Grid • If you have an account on an iRODS data grid, find • your account name and password • the environment file (.irodsEnv) that goes with it • Make directory .irods/ in your home directory > mkdir ~/.irods • Copy the .irodsEnv file into ~/.irods; edit if necessary to insert your user name, zone name, default resource, etc • This will direct your client to the intended data grid, as the intended user.

10. Sample .irodsEnv file (partial) # iRODS server host name: irodsHost ‘tutoricat.renci.org.renci.org' # iRODS server port number: irodsPort 1247 # Default storage resource name: irodsDefResource ’resc1' # Home directory in iRODS: irodsHome '/centosgrid/home/leesa' # Current directory in iRODS: irodsCwd '/centosgrid/home/leesa' # Account name: irodsUserName ’leesa' # Zone: irodsZone ’centosgrid’ … In this example, user name is “leesa” iRODS data grids are referred to as “zones”

11. Some iRODS Clients • iDrop web – iDrop, iDrop-lite http://iren-web.renci.org:8080/idrop-web/login/login • PHP web browser http://iren-web.renci.org/rodsweb • icommands – Unix client https://www.irods.org/index.php/icommands • FUSE (Filesystem in Userspace) client https://www.irods.org/index.php/iRODS_FUSE • Many others supplied by user communities

12. Unix client: icommands See https://wiki.irods.org/index.php/icommands for the complete list and full documentation. Unix-like ils ipwd icd ichmod irm imkdir • FTP-like • iinit • iexit • iput • iget • ipasswd • irsync • ichksum • imv • icp • ienv These slides do not contain an exhaustive list of the icommands.

13. icommands(continued) • Functional ireg ibun irepl Metadata-related imeta iquest idbo Informational ienv ilsresc iuserinfo ihelp – list all icommands • Rule-oriented irule iqstat iqdel iqmod idbug The –h option to any icommand gives documentation on that command.

14. icommands - examples This will store scrambled pw in ~/.irods/.irodsA > iinit Enter your current iRODS password: ***** > ienv NOTICE: Release Version = rods4.0.0, API Version = d NOTICE: irodsHost=tutoricat.renci.org NOTICE: irodsPort=1247 NOTICE: irodsDefResource=resc1 NOTICE: irodsHome=/centosgrid/home/leesa NOTICE: irodsCwd=/centosgrid/home/leesa NOTICE: irodsUserName=leesa NOTICE: irodsZone=centosgrid … > iexit full View your environment This deletes the ~/.irods/.irodsA authentication file.

15. icommands - examples See your current working directory. > ipwd /centosgrid/home/leesa > ils /centosgrid/home/leesa: irods-resource-4.0.0-64bit.deb C- /centosgrid/home/leesa/slides Listing the contents of your current directory. This entry is a collection. Directories in iRODS are called collections. Collection naming convention: /zone_name/home/user_name/collection_name

16. icommands – to ingest and retrieve data > iput –K notes (-K: to calculate and store checksums) > iput iquest-attributes (no checksums) > ils /centosgrid/home/leesa: iquest-attributes irods-resource-4.0.0-64bit.deb notes C- /centosgrid/home/leesa/slides > iget –k irods-resource-4.0.0-64bit.deb (-k: to verify checksum without storing) > irm –f notes (-f to force the remove, physically. Without –f, the file is simply moved to the Trash collection on the same resource.) -k and –K options for checksum calculation and comparison

17. icommands - examples Long listing of my current directory > ils -L /centosgrid/home/leesa: leesa 0 demoResc 1883 2014-05-01.13:58 & iquest-attributes demoResc generic /var/lib/irods/iRODS/Vault/home/leesa/iquest-attributes leesa 0 resc1 28937358 2014-04-30.14:47 & irods-resource-4.0.0-64bit.deb 02b7dd912af6e615f4680594fe3e2dc8 resc1 generic /var/irodsVault/home/leesa/irods-resource-4.0.0-64bit.deb C- /centosgrid/home/leesa/slides Shows owner, resource, file, checksum (if calculated), physical path, …

18. ilsresc – list available resources See the resources available on your data grid On centosgrid: > ilsresc resc1 demoResc resc2 Resources can share hosts but typically don’t. Each resource has its own unique physical location (path).

19. icommands – ACLs (access control lists) and ichmod > ils -A /centosgrid/home/leesa: ACL - leesa#centosgrid:own Inheritance - Disabled iquest-attributes ACL - leesa#centosgrid:own irods-resource-4.0.0-64bit.deb ACL - leesa#centosgrid:own C- /centosgrid/home/leesa/slides • > ichmod read guest iquest-attributes • > ils –A • /centosgrid/home/leesa: • ACL - leesa#centosgrid:own • Inheritance - Disabled • iquest-attributes • ACL - leesa#centosgrid:own guest#centosgrid:read object • irods-resource-4.0.0-64bit.deb • ACL - leesa#centosgrid:own • C- /centosgrid/home/leesa/slides

20. ichmod and group “public” Every user in the data grid is a member of user group “public” > ichmod -r read public slides > ils -A slides /centosgrid/home/leesa/slides: ACL - leesa#centosgrid:own g:public#centosgrid:read object Inheritance - Disabled irods4.0-intro.pptx ACL - public#centosgrid:read object leesa#centosgrid:own “ichmod –h” for help

21. irepl – replicating data objects a replica ≠ a copy Replicating does not create a newlogical object; copying does. > ils /centosgrid/home/rods: hello > irepl -R resc1 hello > ils /centosgrid/home/rods: hello > ils -L /centosgrid/home/rods: rods 0demoResc22 2014-04-16.12:28 & hello demoResc generic /var/lib/irods/iRODS/Vault/home/rods/hello rods 1resc122 2014-05-01.17:03 & hello resc1 generic /var/irodsVault/home/rods/hello The collection doesn’t change with replication of data object “hello” which is still the same logical object as before. Use the long listing to see all replicas and multiple physical locations

22. irepl • See https://irods.sdsc.edu/index.php/irepl for complete list of options • Some options -B: Backup mode - if a good copy already exists in selected resource, don't make another copy -P: show progress -r: recursive replication -R: destination resource -X restart_filename: turn on restart option and accumulate restart info into restart_filename “irepl –h” for help

23. ireg – register data into iRODS In-place ingestion of data into iRODS Ingest data into iRODS without making an additional copy and with no data transfer Example • Unix directory/projects/state-data (not an iRODS collection) contains 500TB of state LiDAR data. • We want to manage the LiDAR data with iRODS. • How to ingest the data into iRODS without copying or transferring 500TB?! continued… “ireg –h” for help

24. ireg example (continued) Set-up • /projects/state-data must be mounted on the iRODS server host • Data admin sets up /projects/state-data as an iRODS resource: > iadmin mkresc stateResc unixfilesystem tutorres.renci.org:/projects/state-data Creating resource: Name: "stateResc" Type: "unixfilesystem" Host: "tutorres.renci.org" Path: "/projects/state-data" Context: "" continued … stateResc is now in listed as a resource: > ilsresc resc1 demoResc resc2 stateResc

25. ireg example (continued) Usage • Data admin registers existing data object or collection into iRODS: > ireg -C -R stateResc/projects/state-data/ /centosgrid/home/state-data/ > ils /centosgrid/home/state-data /centosgrid/home/state-data: lidar-d23720431300go20040824.asc Register incoming files rigorously OR modify a directory only through iRODS once it has been registered to keep the iCAT consistent with the directory. /centosgrid/home/state-data/ Is now an iRODS collection contained in stateResc If new files are added to the directory, ireg can be run again, with option –f, to pick up as-yet unregistered files.

26. ibun – for bundling files Tar up and expand files for efficient iput/iget • iput a tarball and expand it within iRODS > tar -chlf tutorials.tar -C irods-4.0 . > iput -Dtar tutorials.tar > ibun -x tutorials.tar irods-4.0 Unix command to tar up a directory icommand > ils /centosgrid/home/leesa: iquest-attributes irods-resource-4.0.0-64bit.deb notes tutorials.tar icommand > ils /centosgrid/home/leesa: iquest-attributes irods-resource-4.0.0-64bit.deb notes tutorials.tar C- /centosgrid/home/leesa/irods-4.0 C- /centosgrid/home/leesa/slides

27. ibun – for bundling files Tar up and expand files for efficient iput/iget • Tar up files in iRODS for iget > ibun -cDtar slides.tar slides > iget slides.tar > tar –xvf slides.tar –C slides icommand > ils /centosgrid/home/leesa: iquest-attributes irods-resource-4.0.0-64bit.deb notes slides.tar tutorials.tar C- /centosgrid/home/leesa/irods-4.0 C- /centosgrid/home/leesa/slides icommand Unix “ibun –h” for help

28. iquest – querying the iCAT • To see attributes that can be queried: > iquest attrs • See https://wiki.irods.org/index.php/iquest for examples. • > iquest "SELECT DATA_PATH, DATA_NAME, DATA_CHECKSUM WHERE DATA_RESC_NAME like 'resc2'” DATA_PATH = /opt/irodsVault/home/leesa/notes DATA_NAME = notes DATA_CHECKSUM = 0ac4e079aa230d87353fec33f1ec71cd ------------------------------------------------------------ DATA_PATH = /opt/irodsVault/home/rods/yum-fuse-libraries-notes DATA_NAME = yum-fuse-libraries-notes DATA_CHECKSUM = ------------------------------------------------------------ (Checksums displayed only when they have already been calculated and stored, of course)

29. iquest – querying the iCAT • Useful when you want to remove a resource and you discover it isn’t empty: > iquest "SELECT DATA_NAME, USER_NAME, COLL_NAME WHERE DATA_RESC_NAME like ‘tempResc'" DATA_NAME = slide-list-html USER_NAME = rods COLL_NAME = /centosgrid/trash/home/rods ----------------------------------------------------------------- • Administrator can add SQL strings to be invoked by users of the data grid • Data grid-specific queries (added by admin) > iquest --sql 'pre-defined SQL string' [format] [arguments] A file was removed with irm but not physically deleted (moved to Trash on the resource)

30. imeta – add, view, modify metadata • > imeta add –d hello “Date” “2 february 2012” • > imeta ls –d hello AVUs defined for dataObj hello: attribute: Meta1 value: hello units: ---- attribute: Date value: 2 february 2012 units: • > imeta rm –d hello “Meta1” “hello” “imeta –h” for help

31. iadmin – administrative functions • h for help, “h <command>” for help for that command • q to exit • Add new users, modify passwords, add new resources, federate to remote zones, create resource groups,… • mkresc/rmresc • mkuser/rmuser, moduser (modify passwords) • mkzone/rmzone, modzone (for federation) • Information on users, resources, tokens, etc • lt (el-tee), “lt <token type>” for listing of that token’s possible values • lu, lr, lz,…

32. Cloud Resources • Admin user must set up the cloud resource, for example: • S3 Resource: s3Resc • This is just another iRODS resource for the users, who can manage their cloud data just as all other iRODS data: > iput –K –R s3Resc my_file (put data into the cloud resource) > irepl –R s3Resc another_file (replicate into the cloud resource) > ichmod read public my_file (give public access to data in the cloud) > ichmod write user1 my_collection (give user1 permission to write to a collection in the cloud)

33. Realizable Objects – coming in 4.1??? • Typical iRODS objects contain data • Realizable objects: • symbolic links to iRODS objects • symbolic links to external data sources • workflow procedures to regenerate data • Implemented in community iRODS, still to come for the merged release (planned for 4.1)

34. Database Resources???? • Database Resource (DBR): a database, queried and updated via SQL (or other, for non-SQL) • Database object (DBO): an interface to a DBR, typically a (SQL) query that returns results • iRODS agent will open and close DB as needed; results of the query are directly returned to user • Query results are stored to an iRODS data object, a DBO Results file (DBOR). • iRODS access controls are applied on the DBR and DBO.

35. Microservices and Rules Microservices • C code • the unit of work within iRODS • called by rules – can only be run by a rule Rules • execute the microservices • compose microservices and other rules into workflows

36. Rules • Rules can run workflows of microservices (and other rules) • Implement computer actionable policies • Provide implementation of data policies • Retention, distribution, arrangement • Authenticity, provenance, description • Integrity, replication, synchronization • Deletion, trash cans, versioning • Archiving, staging, caching • Authentication, authorization, sharing • Access, approval, IRB, audit trails, report generation • Assessment criteria, validation • Derived data product generation, format parsing

37. Running Rules • Triggered by events/policy points • Contained in the (distributed) rule base: • /etc/irods/core.re • many rules with the same name; the first rule with satisfied condition is executed; others are skipped • Can be run with irule: manual execution • Delayed execution • iqstat • iqdel

38. irule – to run a rule manually • Example rules to tweak and run in the software distribution iRODS/clients/icommands/test/rules3.0 • irule -F rulemsiListEnabledMS.r – list enabled microservices Works with 4.0.1, not 4.0 • irule -F rulemsiAdmShowCoreRE.r -can only be run by admin users • NB: not all the test rules in iRODS/clients/icommands/test/rules3.0 are running yet in 4.0.