1 / 56

iRODS Tutorial Rules and Microservices

iRODS Tutorial Rules and Microservices. iRODS Rules and Microservices. Introduction to Rules and Microservices Simple Rules and Database Queries Complex Rules and Scheduling. iRODS Microservices Documentation. Available from Amazon

maxime
Download Presentation

iRODS Tutorial Rules and Microservices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. iRODS TutorialRules and Microservices

  2. iRODS Rules and Microservices • Introduction to Rules and Microservices • Simple Rules and Database Queries • Complex Rules and Scheduling

  3. iRODS Microservices Documentation Available from Amazon The integrated Rule-Oriented Data System (iRODS) Micro-service Workbook http://www.amazon.com/dp/1466469129 Contains example rules for running each microservice. On-line documentation https://www.irods.org/doxygen/

  4. I. Introduction to Rules and Microservices

  5. Rules • See https://www.irods.org/index.php/Changes_and_Improvements_to_the_Rule_Language_and_the_Rule_Engine • Rules implement data policy by running (workflows of) microservices • Retention, distribution, arrangement • Authenticity, provenance, description • Integrity, replication, synchronization • Deletion, trash cans, versioning • Archiving, staging, caching • Authentication, authorization, redaction • Access, approval, IRB, audit trails, report generation • Assessment criteria, validation • Derived data product generation, format parsing Policy is the clear statement of how data will be managed over its life cycle.

  6. Microservices • C code • The unit of work within iRODS • Run by rules • Composed into workflows by rules

  7. Running Rules • Rules triggered by events/policy points are contained in the (distributed) rule base: • iRODS/server/config/reConfigs/core.re • conditional execution • first rule with satisfied condition is executed; others are skipped • Can be run with irule - manual execution • Delayed execution • iqstat • iqmod

  8. irule – to run a rule manually • Example rules to tweak and run are in the software distribution at iRODS/clients/icommands/test/rules3.0 • irule -F rule_file_name.r • Some rules can only be run by admin users

  9. An example – list all microservices • Download listMS.r from /snicZone/home/pracerods/rules in iRODS • listMS.r (lists all available microservices) ListAvailableMS { msiListEnabledMS(*KVPairs); writeKeyValPairs("stdout", *KVPairs, ": "); } INPUT null OUTPUT ruleExecOut • Run it with irule: irule –F listMS.r

  10. Format of a Rule Rule_name{ microservice1(…,*A,…,*B); microservice2(*A,…); } INPUT *A=”first_input", *B=”second_input" OUTPUT ruleExecOut OR Rule_name(*arg) { on(exp) { microservice1(…,*arg,*C); microservice2(*C,…); } } INPUT null OUTPUT ruleExecOut *A and *B are workflow variables. ruleExecOut accesses the internal ruleExecInfo (rei) structure managed by iRODS. • A rule can take arguments. • A rule can be executed conditionally. • Use “null” if there are no input parameters.

  11. Rules and Parameters • Literals • constants: strings or numbers • a variable name not beginning with a special character (#, $ or *) is taken as string input • can only be used as input parameters (not output) • Workflow variables • Session state variables • Persistent state variables

  12. Workflow Variables (*variables) • For example, in the following workflow chain: myRule{ msiDataObjOpen(*file,*FD); msiDataObjRead(*FD,10000,*BUF); writeLine(“stdout”,*BUF); … } INPUT *file=“/compZone/home/leesa/hello” OUTPUT ruleExecOut (‘stdout’ is a structure managed by iRODS.) • *fileis an input parameter • *FD is output from msiDataObjOpen and input to msiDataObjRead. • *file, *FD, and *BUF are workflow variables

  13. Session Variables ($variables) • contain temporary information maintained during a server session • contain information about client-server connection, data objects, user information, resource information, etc. • contain information that can be sent back to the client. Example: stdout, stderr • persistent across rule executions in the same session, so can be used to pass information between rule executions • pre-defined by iRODS, stored as a complex C-structure (the rei structure). See https://www.irods.org/index.php/RuleExecInfo_Structure_%26_$-variables

  14. Session Variables ($variables) • $variables map to specific locations in the REI structure - mapping contained in server/config/reConfigs/core.dvm • Example: $objPath||rei->doi->objPath $objPath||rei->doinp->objPath $dataType||rei->doi->dataType $userNameClient||rei->uoic->userName $collName||rei->coi->collName $collParentName||rei->coi->collParentName • https://www.irods.org/index.php/Session_State_Variables (Mappings are not necessarily unique.)

  15. Persistent State Variables (#variables) • See iRODS Primer • “iquest” uses these field names: iquest attrs • https://www.irods.org/index.php/Persistent_State_Information_Variables • iRODS/lib/core/include/rodsGenQuery.h defines the attributes available via the General Query interface. • Names begin with ‘COL_’ (column) for easy identification in the source code; mapping to variables without COL_ contained in iRODS/lib/core/include/rodsGenQueryNames.h

  16. Persistent State Information ZONE_ID ZONE_NAME ZONE_TYPE ZONE_CONNECTION ZONE_COMMENT ZONE_CREATE_TIME ZONE_MODIFY_TIME USER_ID USER_NAME USER_TYPE USER_ZONE USER_DN USER_INFO USER_COMMENT USER_CREATE_TIME USER_MODIFY_TIME RESC_ID RESC_NAME RESC_ZONE_NAME RESC_TYPE_NAME RESC_CLASS_NAME RESC_LOC RESC_VAULT_PATH RESC_FREE_SPACE RESC_FREE_SPACE_TIME RESC_INFO RESC_COMMENT RESC_CREATE_TIME RESC_MODIFY_TIME RESC_STATUS DATA_ID DATA_COLL_ID DATA_NAME DATA_REPL_NUM DATA_VERSION DATA_TYPE_NAME DATA_SIZE DATA_RESC_GROUP_NAME DATA_RESC_NAME DATA_PATH DATA_OWNER_NAME DATA_OWNER_ZONE DATA_REPL_STATUS DATA_STATUS DATA_CHECKSUM DATA_EXPIRY DATA_MAP_ID DATA_COMMENTS DATA_CREATE_TIME DATA_MODIFY_TIME DATA_ACCESS_TYPE DATA_ACCESS_NAME DATA_TOKEN_NAMESPACE DATA_ACCESS_USER_ID DATA_ACCESS_DATA_ID COLL_ID COLL_NAME COLL_PARENT_NAME COLL_OWNER_NAME COLL_OWNER_ZONE COLL_MAP_ID COLL_INHERITANCE COLL_COMMENTS COLL_CREATE_TIME COLL_MODIFY_TIME COLL_ACCESS_TYPE COLL_ACCESS_NAME COLL_TOKEN_NAMESPACE COLL_ACCESS_USER_ID COLL_ACCESS_COLL_ID META_DATA_ATTR_NAME META_DATA_ATTR_VALUE META_DATA_ATTR_UNITS META_DATA_ATTR_ID META_DATA_CREATE_TIME

  17. Persistent State Information META_DATA_MODIFY_TIME META_COLL_ATTR_NAME META_COLL_ATTR_VALUE META_COLL_ATTR_UNITS META_COLL_ATTR_ID META_NAMESPACE_COLL META_NAMESPACE_DATA META_NAMESPACE_RESC META_NAMESPACE_USER META_RESC_ATTR_NAME META_RESC_ATTR_VALUE META_RESC_ATTR_UNITS META_RESC_ATTR_ID META_USER_ATTR_NAME META_USER_ATTR_VALUE META_USER_ATTR_UNITS META_USER_ATTR_ID RESC_GROUP_RESC_ID RESC_GROUP_NAME USER_GROUP_ID USER_GROUP_NAME RULE_EXEC_ID RULE_EXEC_NAME RULE_EXEC_REI_FILE_PATH RULE_EXEC_USER_NAME RULE_EXEC_ADDRESS RULE_EXEC_TIME RULE_EXEC_FREQUENCY RULE_EXEC_PRIORITY RULE_EXEC_ESTIMATED_EXE_TIME RULE_EXEC_NOTIFICATION_ADDR RULE_EXEC_LAST_EXE_TIME RULE_EXEC_STATUS TOKEN_NAMESPACE TOKEN_ID TOKEN_NAME TOKEN_VALUE TOKEN_VALUE2 TOKEN_VALUE3 TOKEN_COMMENT AUDIT_OBJ_ID AUDIT_USER_ID AUDIT_ACTION_ID AUDIT_COMMENT AUDIT_CREATE_TIME AUDIT_MODIFY_TIME SL_HOST_NAME SL_RESC_NAME SL_CPU_USED SL_MEM_USED SL_SWAP_USED SL_RUNQ_LOAD SL_DISK_SPACE SL_NET_INPUT SL_NET_OUTPUT SL_CREATE_TIME SLD_RESC_NAME SLD_LOAD_FACTOR SLD_CREATE_TIME

  18. Rule Condition • Boolean expression • Examples • Run if msiService succeeds: rule1 { on (msiService >= 0) { ... } } • Run if resource is demoResc8: rule2{ on ($rescName == demoResc8) {…} } • Run if the pathname begins with /x/y/z: Rule3{ on ($objPath like /x/y/z/*) {…} } • Same rule can give different actions depending on which condition is met • Many operators ==, !=, >, <, >=, <= %%, !! (and, or) expr like reg-expr , expr not like reg-expr , expr ::= string

  19. Policy Enforcement Points Rule Triggers – see core.re • These are locations within iRODS framework where an event or state (of the environment) prompts a rule to execute • An action (eg, an icommand) may involve multiple policy enforcements points • Policy enforcement points • Pre-action policy (eg, selection of storage location) • Execution/action policy (eg, file deletion) • Post-action policy (eg, create secondary data products) • Actions (the triggered rules) and policy enforcement points are contained in iRODS/server/config/reConfigs/core.re

  20. core.re • Take a look at iRODS/server/config/reConfigs/core.re • Contains rules corresponding to a number of events • Create, delete, modify users • Open, create, put, copy, replicate, delete files • Create, remove collections • Create/modify metadata • Contains some rules to set general policy • ACL policy (strict or not) • Public user policy • Default resource policy • An empty rule corresponds to a trigger point for which no action has been defined

  21. Some Policy Enforcement Points ACTION acCreateUser acDeleteUser acGetUserbyDN acTrashPolicy acAclPolicy acSetCreateConditions acDataDeletePolicy acRenameLocalZone acSetRescSchemeForCreate acRescQuotaPolicy acSetMultiReplPerResc acSetNumThreads acVacuum acSetResourceList acSetCopyNumber acVerifyChecksum acCreateUserZoneCollections acDeleteUserZoneCollections acPurgeFiles acRegisterData acGetIcatResults acSetPublicUserPolicy acCreateDefaultCollections acDeleteDefaultCollections PRE-ACTION POLICY acPreProcForCreateUser acPreProcForDeleteUser acPreProcForModifyUser acPreProcForModifyUserGroup acChkHostAccessControl acPreProcForCollCreate acPreProcForRmColl acPreProcForModifyAVUMetadata acPreProcForModifyCollMeta acPreProcForModifyDataObjMeta acPreProcForModifyAccessControl acPreprocForDataObjOpen acPreProcForObjRename acPreProcForCreateResource acPreProcForDeleteResource acPreProcForModifyResource acPreProcForModifyResourceGroup acPreProcForCreateToken acPreProcForDeleteToken acNoChkFilePathPerm acPreProcForGenQuery acSetReServerNumProc acSetVaultPathPolicy POST-ACTION POLICY acPostProcForCreateUser acPostProcForDeleteUser acPostProcForModifyUser acPostProcForModifyUserGroup acPostProcForDelete acPostProcForCollCreate acPostProcForRmColl acPostProcForModifyAVUMetadata acPostProcForModifyCollMeta acPostProcForModifyDataObjMeta acPostProcForModifyAccessControl acPostProcForOpen acPostProcForObjRename acPostProcForCreateResource acPostProcForDeleteResource acPostProcForModifyResource acPostProcForModifyResourceGroup acPostProcForCreateToken acPostProcForDeleteToken acPostProcForFilePathReg acPostProcForGenQuery acPostProcForPut acPostProcForCopy acPostProcForCreate

  22. Out-of-the-Box Services Microservices for… • Queries on metadata catalog • Interaction with web services • Invocation of external applications • Workflow constructs (loops, conditionals, exit) • Remote and delayed execution control

  23. Microservices msiGetValByKey msiAddKeyVal assign ifExec break applyAllRules msiExecStrCondQuery msiExecStrCondQueryWithOptions msiExecGenQuery msiMakeQuery msiMakeGenQuery msiGetMoreRows msiAddSelectFieldToGenQuery msiAddConditionToGenQuery msiPrintGenQueryOutToBuffer msiExecCmd msiSetGraftPathScheme msiSetRandomScheme msiCheckHostAccessControl msiGetIcatTime msiGetTaggedValueFromString msiXmsgServerConnect msiXmsgCreateStream msiCreateXmsgInp msiSendXmsg msiRcvXmsg msiXmsgServerDisConnect msiString2KeyValPair msiStrArray2String msiRdaToStdout msiDataObjCreate msiDataObjOpen msiDataObjClose msiDataObjLseek msiDataObjRead msiDataObjWrite msiDataObjUnlink msiDataObjRepl msiDataObjCopy msiExtractNaraMetadata msiSetMultiReplPerResc msiAdmChangeCoreIRB msiAdmShowIRB msiAdmShowDVM msiAdmShowFNM msiAdmAppendToTopOfCoreIRB msiAdmClearAppRuleStruct msiAdmAddAppRuleStruct msiGetObjType msiAssociateKeyValuePairsToObj msiExtractTemplateMDFromBuf msiReadMDTemplateIntoTagStruct msiDataObjPut msiDataObjGet msiDataObjChksum msiDataObjPhymv msiDataObjRename msiDataObjTrim msiCollCreate print_hello_arg msiVacuum msiQuota msiGoodFailure msiSetResource msiCheckPermission msiCheckOwner msiCreateUser msiCreateCollByAdmin msiSendMail recover_print_hello msiCommit msiRollback msiDeleteCollByAdmin msiDeleteUser msiAddUserToGroup msiSetDefaultResc msiSetRescSortScheme msiSysReplDataObj msiStageDataObj msiSetDataObjPreferredResc msiSetDataObjAvoidResc msiSortDataObj msiSysChksumDataObj msiSetDataTypeFromExt msiSetNoDirectRescInp msiSetNumThreads msiDeleteDisallowed msiOprDisallowed msiRmColl msiReplColl msiCollRepl msiPhyPathReg msiObjStat msiDataObjRsync msiFreeBuffer msiNoChkFilePathPerm msiNoTrashCan msiSetPublicUserOpr whileExec forExec delayExec remoteExec forEachExec msiSleep writeString writeLine writeBytesBuf writePosInt writeKeyValPairs msiGetDiffTime msiGetSystemTime msiHumanToSystemTime msiStrToBytesBuf msiApplyDCMetadataTemplate msiListEnabledMS msiSendStdoutAsEmail msiPrintKeyValPair

  24. Microservices msiRdaToDataObj msiRdaNoResults msiRdaCommit msiAW1 msiRdaRollback msiRenameLocalZone msiRenameCollection msiAclPolicy msiRemoveKeyValuePairsFromObj msiDataObjPutWithOptions msiDataObjReplWithOptions msiDataObjChksumWithOptions msiDataObjGetWithOptions msiSetReServerNumProc msiGetStdoutInExecCmdOut msiGetStderrInExecCmdOut msiAddKeyValToMspStr msiPrintGenQueryInp msiTarFileExtract msiTarFileCreate msiPhyBundleColl msiWriteRodsLog msiServerMonPerf msiFlushMonStat msiDigestMonStat msiSplitPath msiGetSessionVarValue msiAutoReplicateService msiDataObjAutoMove msiGetContInxFromGenQueryOut msiSetACL msiSetRescQuotaPolicy msiPropertiesNew msiPropertiesClear msiPropertiesClone msiPropertiesAdd msiPropertiesRemove msiPropertiesGet msiPropertiesSet msiPropertiesExists msiPropertiesToString msiPropertiesFromString msiRecursiveCollCopy msiGetDataObjACL msiGetCollectionACL msiGetDataObjAVUs msiGetDataObjPSmeta msiGetCollectionPSmeta msiGetDataObjAIP msiLoadMetadataFromDataObj msiExportRecursiveCollMeta msiCopyAVUMetadata msiGetUserInfo msiGetUserACL msiCreateUserAccountsFromDataObj msiLoadUserModsFromDataObj msiDeleteUsersFromDataObj msiLoadACLFromDataObj msiGetAuditTrailInfoByUserID msiGetAuditTrailInfoByObjectID msiGetAuditTrailInfoByActionID msiGetAuditTrailInfoByKeywords msiGetAuditTrailInfoByTimeStamp msiSetDataType msiGuessDataType msiMergeDataCopies msiIsColl msiIsData msiGetCollectionContentsReport msiGetCollectionSize msiStructFileBundle msiCollectionSpider msiFlagDataObjwithAVU msiFlagInfectedObjs

  25. Microservice Modules • A module must be compiled with the code, though pluggable modules (coming) will no longer require recompiling the whole code • Consult the Microservice book: The integrated Rule-Oriented Data System (iRODS) Micro-service Workbook to see which module a microservice is contained in • Enable that module: Enabled… yes in info.txt • Example • > irule -F rulemsiCopyAVUMetadata.r ERROR: rcExecMyRule error. status = -1102000 NO_MICROSERVICE_FOUND_ERR Level 0: DEBUG: execMicroService3: no micro service found line 12, col 2 msiFlagDataObjwithAVU(*Source,*Flag,*Status); • msiFlagDataObjwithAVU is contained in module ERA • enable module ERA • ./irodsctl istop • ./irodssetup

  26. Creating New MicroservicesModules Define Function Siganture Register mService Mapping Create Internal Function Describe mService Any function can be converted into a microservice, but it’s important to implement recovery microservices Important!! Implement recovery mService This procedure will change with coming releases.

  27. II. Simple Rules and Database Queries

  28. Example Rules • See iRODS/clients/icommands/test/rules3.0 in the iRODS distribution • See /compZone/home/leesa/rules in compZone data grid • See /compZone/home/rods/rules in compZone data grid • See /snicZone/home/pracerods/rules in snicZone • See http://www.renci.org/~leesa/rules/

  29. Listing the Rule Base showCore.r showCoreRules { # Listing of the core.re file # # Input parameters: # none msiAdmShowCoreRE(); } INPUT null OUTPUT ruleExecOut An admin user can execute the rule to show the rule base: • irule –vF showCore.r

  30. rulemsiExecCmd.rRun an executable script myTestRule { #Input parameters are: # Command to be executed - located in directory irods/server/bin/cmd # Optional command argument string # Optional host address for command execution # Optional hint for remote data object path, command is executed on host # where the file is stored # Optional flag. If > 0, use the resolved physical data object path as first argument #Output parameter is: # Structure holding status, stdout, and stderr from command execution #Output: # Command result is: “Hello world written from irods” # msiExecCmd(*Cmd,*Arg,"null","null","null",*Result); msiGetStdoutInExecCmdOut(*Result,*Out); writeLine("stdout","Command result is"); writeLine("stdout","*Out"); } INPUT *Cmd=”hello", *Arg="written" OUTPUT ruleExecOut “hello” is an executable script in iRODS/server/bin/cmd.

  31. Example Policy ImplementationUsing “asPostProcForPut” to implement policy, depending on resource Data coming in to a target iRODS resource triggers a script that takes some desired action, triggers message to admin (unix) user acPostProcForPut{ on($rescName like ”uncResc") { writeLine("serverLog","USER, OBJPATH, and FILEPATH: $userNameClient, $objPath and $filePath"); msiExecCmd("resource-trigger.sh","$rescName $objPath $userNameClient","null","null","null",*Out); msiSendMail("leesa@renci.org","resource $rescName","User $userNameClient just ingested file $objPath into $rescName."); } } • acPostProcForPut is contained in • iRODS/server/config/reConfigs/core.re • resource-trigger.sh is contained in /server/bin/cmd • and must be executable!!

  32. Example script resource-trigger.sh • > more resource-trigger.sh #!/bin/sh # echo "execCmdRule: "$execCmdRule rescName=$1 objPath=$2 userNameClient=$3 echo "User $userNameClient just ingested file $objPath into $rescName" echo "User $userNameClient just ingested file $objPath into $rescName" > /tmp/resource.out

  33. Example Policy ImplementationUsing asPostProcForPut to implement policy: inputs to a specific resource Data coming in to a target iRODS collection triggers a script that takes some desired action (sending data to a remote ftp site) acPostProcForPut{ on($objPath like ”/snicZone/home/prace-class/*") { writeLine("serverLog", "$userNameClient sending $objPath to NCDC."); msiSplitPath($filePath,*fileDir,*fileName); msiExecCmd(“outgoing.sh","*fileDir *fileName","null","null","null",*Out); msiSendMail("leesa@renci.org", "send to NCDC","User $userNameClient sent $objPath to NCDC."); } } • acPostProcForPut is contained in • iRODS/server/config/reConfigs/core.re • acPostProcForPut is the same rule in both examples! Just using different conditions.

  34. Example script outgoing.sh • > more outgoing.sh #!/bin/sh HOST=ftp.****.****.*** USER=anonymous PASSWD=leesa@renci.org srcDir=$1 srcFile=$2 echo $srcDir echo $srcFile #echo "Place holder for outgoing script. Dir: $srcDir, File: $srcFile" echo "Place holder for outgoing script. Dir: $srcDir, File: $srcFile" > /tmp/test.out

  35. iquest – for constrained querying of the iCAT • “iquest attrs” shows the attributes you can query on • Query iCAT of remote zoneA: iquest –z zoneA … Example: User rods#compZone, logged into compZone, gives the command > iquest -z snicZone "SELECT COLL_NAME, DATA_NAME WHERE COLL_NAME like '/snicZone/home/rods%'" Zone is snicZone COLL_NAME = /snicZone/home/rods#compZone DATA_NAME = hello ------------------------------------------------------------ The response comes from the snicZone iCAT, not the user’s home compZone iCAT.

  36. More on iquest Set SQL logging to see actual SQL queries generated using iquest • Edit scripts/perl/irodsctl.pl - uncoment the line $spLogSql = "1”; • ./irodsctl irestart • Logged into iRODS/server/log files

  37. Specific QueriesMore on iquest • iadmin asq – add SQL query Example: show all data files with metadata, along with the metadata > iadmin asq 'select object_id, data_id, data_name, meta_attr_name, meta_attr_value from r_objt_metamap, r_meta_main, r_data_main where r_meta_main.meta_id = r_objt_metamap.meta_id and object_id = data_id;' file-metadata Now any user can make this query using iquest: > iquest –-sql file-metadata Data grid administrators: beware what you enable with specific queries; the iCAT contains passwords, ACLs, etc. Control carefully the access you give. Alias to the query

  38. GenQuery • The primary query interface in IRODS • msiMakeGenQuery constructs a SQL string that’s passed to the iCAT by a subsequent call to msiExecGenQuery • Sets up a data structure, so use msiGetValByKey to get values out of the call to msiExecGenQuery • Many iRODS commands (eg, ils) use genQuery behind the scenes • Generates SQL to perform the query subject to access control • Simplified SQL-like interface – user doesn’t need to know the iCAT schema to make queries • Queries on same attributes as iquest

  39. GenQuery Example test0.r – list all data objects with the name “hello” mygenQuery{ # Input: # string containing attribute list to select on # string containing condition # Output: # GenQInp: an internal data structure # msiMakeGenQuery("COLL_NAME", *Condition, *GenQInp); msiExecGenQuery(*GenQInp,*GenQOut); foreach(*GenQOut){ msiGetValByKey(*GenQOut,"COLL_NAME",*outcoll); writeLine("stdout", "*outcoll/hello"); } } INPUT *Condition="DATA_NAME = 'hello'" OUTPUT ruleExecOut

  40. GenQuery Example irule –F test0.r /compZone/home/leesa/hello /compZone/home/outgoing/hello /compZone/home/rods/hello /compZone/trash/home/rods/hints/linux-hints/ischia2-hints/hello This rule was run as user rods (admin user) on compZone. GenQuery will not allow all users to make all queries. ACL permissions are respected with GenQuery. Examples: see /compZone/home/rods/rules/genQuery or /snicZone/home/pracerods/rules/genQuery Try tweaking and running some examples.

  41. Database Resources • DataBase Resource (DBR): an external database (or similar tabular information) that can be queried and updated via SQL statements (or other, for non-SQL) • Database object (DBO): an interface to the DBR - typically a query that returns results • DBOs typically contain SQL • Query results can be stored to an iRODS data object, a DBO Results file (DBOR). • idbo command – to run the DBO query • access controls imposed by iRODS ACLs • https://www.irods.org/index.php/Database_Resources and https://www.irods.org/index.php/Database_Resource_Administration

  42. Setting Up a DBR • See https://www.irods.org/index.php/Database_Resource_Administration • Install a non-IES server (an additional resource) on a machine that has access to the remote DB. Example: install on iren.renci.org. • On iren.renci.org, edit server/config/dbr.config - add a line containing: DBR-name DBMS-username DBMS-password DB-type (postgres, oracle, mysql) Example: leesabase leesa leesapw postgresql • On iren.renci.org, put a .odbc.ini file into $HOME [IRODS_DBR_leesabase] Driver=/home/leesa/tutorial-testing/pgsql/lib/libodbcpsql.so.2.0.0 Debug=0 CommLog=0 Servername=informatics.renci.org Database=leesabase ReadOnly=no Ksqo=0 Port=5432 (Can copy – mostly – from $HOME/.odbc.ini on your data grid server.)

  43. Setting Up a DBR • Edit iRODS/config/config.mk: • uncomment DBR=1 • define POSTGRES_HOME: POSTGRES_HOME=/home/leesa/tutorial-testing/pgsql • Make sure the LD_LIBRARY_PATH includes the postgres libraries: setenv LD_LIBRARY_PATH /home/leesa/tutorial-testing/pgsql/lib • Do a new "make" of the code in iRODS/and then ./irodsctl irestart

  44. Setting Up a DBR • Go over the main data grid (ischia2.renci.org) and use iadmin to define the DBR in the ICAT. See 'iadmin h mkresc’. mkresc Name Type Class Host [Path] (make Resource) Example: mkresc leesabase database postgresql iren.renci.org /home/leesa/tutorial-testing/iRODS3.1/Vault • Give ownership of the DBR to the appropriate user(s) - for write/query permission to the DBR: 'ichmod -R own User-name DBR-name’ Example: ichmod -R own rods leesabase • See iRODS/server/icat/dbr, containing example DBOs. Put a query into a file and ingest into iRODS: iput lt.pg

  45. Setting Up a DBR • Use isysmeta to declare this as a DBO type: isysmeta mod lt.pg datatype 'database object' • Use idbo to execute a query (contained in a dbo) on the DBR (leesabase in this case), running as a user that has ‘own’ permission on the DBR: idbo exec dbrname dboname Example:idbo exec leesabase lt.pg Note: when giving this command, you either need to be in the collection that contains the dbo file, or you need to give the full path name to it.

  46. Using a DBR • idbo to execute a query contained in a DBO • Store results into a DBO Result file (DBOR) by using idbo as an interactive shell: >idbo idbo> output lt.out 'exec' results will be stored in /compZone/home/rods/dbr-examples/lt.out idbo>exec leesabase lt.pg Output written to /compZone/home/rods/dbr-examples/lt.out idbo>quit • Examples: in /compZone/home/rods/dbr-examples NB: For this tutorial the DBR hasn’t been set up on snicZone, so DBOs must be run on compZone.

  47. III. Complex Rules and Scheduling

  48. Delayed Execution • Example myTestRule{ delay("<PLUSET>1m</PLUSET>"){ writeLine("stdout",”Writing message with a delay."); msiSendStdoutAsEmail(*Mailto, "Sending email"); } } INPUT *Mailto="leesa@renci.org" OUTPUT ruleExecOut • Queue management: • iqstat • iqdel • iqmod

  49. Periodic Execution Example myTestRule { # Input parameters are: # Source collection path # Target collection path # Optional target resource # Optional synchronization mode: IRODS_TO_IRODS # Output parameter is: # Status of the operation # Output from running the example is: # Synchronized collection 1 with collection 2 # delay("<PLUSET>5m</PLUSET>EF>1h</EF>"){ msiCollRsync(*srcColl,*destColl,*Resource,"IRODS_TO_IRODS",*Status); writeLine("stdout","Synchronized collection *srcColl with collection *destColl"); } } INPUT *srcColl="/compZone/home/leesa/tutorials", *destColl="/compZone/home/leesa/tutorials2", *Resource="demoResc" OUTPUT ruleExecOut

  50. More Interesting Rules Policy to control where data is stored upon ingestion (courtesy of iPlant) • Data ingested into TACC collections will be forced to resource “corral”: acSetRescSchemeForCreate { ON($objPath like "/tacc/Collections/*") {msiSetDefaultResc(corral,forced);} } (This forces physical location, with no effect whatsoever on logical location.) • All other data goes into iplant resource group, randomly distributed among resources there: acSetRescSchemeForCreate { msiSetDefaultResc("iplantRG","preferred"); msiSetRescSortScheme("random"); } (This means that a default resource doesn’t need to be specified by users.)

More Related