1 / 39

Usage of R-GMA and Grid Application Monitoring

Valeria Ardizzone INFN-Catania 1° EELA Grid School Itacuruçà Island, State of Rio De Janeiro, Brazil. Usage of R-GMA and Grid Application Monitoring. Usage of R-GMA command line interface Usage of R-GMA Web Browser like a Consumer Usage of R-GMA cli tool into jobs

conroy
Download Presentation

Usage of R-GMA and Grid Application Monitoring

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Valeria Ardizzone INFN-Catania 1° EELA Grid School Itacuruçà Island, State of Rio De Janeiro, Brazil Usage of R-GMAandGrid Application Monitoring

  2. Usage of R-GMA command line interface • Usage of R-GMA Web Browser like a Consumer • Usage of R-GMA cli tool into jobs • Usage of R-GMA API to create Producer and Consumer • Usage of Producer and Consumer within application • References Outline

  3. R-GMA command line tool

  4. > voms-proxy-init --voms gilda Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Valeria Ardizzone/Email=valeria.ardizzone@ct.infn.it Enter GRID pass phrase: Creating temporary proxy ............................................................ Done Contacting voms.ct.infn.it:15001[/C=IT/O=GILDA/OU=Host/L=INFN Catania/CN=voms.ct.infn.it/Email=emidio.giorgio@ct.infn.it] "gilda" Done Creating proxy .................................... Done Your proxy is valid until Fri Jun 16 08:20:14 2006 • Before you start the R-GMA command line tool make sure you have a proxy certificate : • Run the command and you should receive the following message on startup: R-GMA Command Line Tool > rgma Welcome to the R-GMA virtual database for Virtual Organisations. ================================================================ Your local R-GMA server is: https://egris198.eela.ufrj.br:8443/R-GMA You are connected to the following R-GMA Registry services: https://egris198.eela.ufrj.br:8443/R-GMA/RegistryServlet You are connected to the following R-GMA Schema service: https://egris198.eela.ufrj.br:8443/R-GMA/SchemaServlet Type "help" for a list of commands. rgma>

  5. Commands are entered by typing at the> rgmaprompt and hitting ‘enter’ to execute the command. • Ahistoryof the commands executed can be accessed using the Up and Down arrow keys. • Tosearch a command from historyuse CTRL-R and type the first few letters of the command to recall. • Command autocompletionis supported (use Tab when you have partly entered a command). Entering Command

  6. help and help <command> Display general help information or specific for a command. • exit or q Exit from R-GMA command line interface. • show …... • set …... • clear history Clear the current session history of commands executed. • write history <filename> Write the session command history to a file. • write results <filename> Write query results to a file. • read <filename> Show all information about the structure of a table General Commands

  7. The command line tool can be used in batch mode in three ways: • rgma –c <command> [ -c <command> …] Executes the command and exit. • rgma –f <filename> Executes commands in file sequentially and exit. • Commands embedded in a shell script: #!/bin/sh $RGMA_HOME/bin/rgma <<EOF set query latest select Service_Name, Status FROM ServiceStatus WHERE Status != 0 EOF Batch mode

  8. Show tables Display the name of all tables existing in the Schema • Describe <tablename> Show all information about the structure of a table • create table Create a table in the R-GMA virtual database schema. • drop table <tablename> Delete a table in the R-GMA virtual database schema. Table Commands

  9. rgma> create TABLE ITATutTable(COD_Test INT PRIMARY KEY, Application VARCHAR(20), Status VARCHAR(10), PercStatus INT, Owner VARCHAR(25)) rgma> show tables Exercise 1: create a table ……. | NetworkUDPPacketLoss | | NetworkFileTransferThroughput | | ITATutTable | rgma> describe ITATutTable +-----------------+------------+-------------------------+----------------+------------------+ | Column name | Type | Primary key | Can be NULL | +-----------------+-------------+------------------------+----------------+------------------+ | COD_Test | INTEGER | Yes | No | | Application | VARCHAR(20) | No | Yes | | Status | VARCHAR(10) | No | Yes | | PercStatus | INTEGER | No | Yes | | Owner | VARCHAR(25) | No | Yes | | MeasurementDate | DATE | No | No | | MeasurementTime | TIME | No | No | +-----------------+------------+-------------------------+----------------+------------------+

  10. Using the command line tool you may work with one producer at a time. • The current producer type can be displayed using: rgma>show producer • Set the latest retention period for tuples published by the producer rgma>set producer latestretentionperiod|lrp <time value> [<units>] • Set the history retention period for the producer. If the producer does not support history queries this command has no effect. rgma>set producer historyretentionperiod|hrp <time value> [<units>] • Producer which handles the INSERT statement. The SQL INSERT statement may be used to add data to the system: rgma> INSERT INTO <Tablename> VALUES (’a’, ’b’, ’c’, ’d’) Producer Properties

  11. rgma> set producer latest Producer type : continuous latest rgma> set producer latestretentionperiod 50 seconds Set producer LRP to 50 seconds rgma> set producer historyretentionperiod 2 minutes Set producer HRP to 2 minutes rgma>describe ITATutTable rgma> insert INTO ITATutTable values(001,'TestProducer','Start',10,'Valeria'); Inserted 1 row into ITATutTable Exercise 2: create a Producer

  12. The behaviour of Consumer varies according to the type of query being executed. In R-GMA there are three basic types of query: LATEST Queries:only the most recent tuple for each primary key HISTORY Queries:all historical tuples for each primary key CONTINUOUS Queries:returns tuples continuously as they are inserted. • The type of query can be changed using the SET QUERY command as follow: rgma> SET QUERY LATEST | CONTINUOUS | HISTORY • The current query type can be displayed using rgma> SHOW QUERY Consumer Properties (1)

  13. The maximum age of tuples to return can also be controlled. To limit the age of latest or historical tuples use the MAXAGE property. rgma> SET MAXAGE <time> seconds|minutes|hours|days • The current maximum tuple age can be displayed using rgma> SHOW MAXAGE • To disable the maximum age, set it to none: rgma> SET MAXAGE none Consumer Properties (2)

  14. The final property affecting queries is timeout. • For a latest or history query the timeout exists to prevent a problem (e.g. network failure) from stopping the query from completing. • For a continuous query, timeout indicates how long the query will continue to return new tuples. Default timeout is 1 minute and it can be changed using rgma>SET TIMEOUT <time> seconds|minutes|hours|days • The current timeout can be displayed using rgma>SHOW TIMEOUT Consumer Properties (3)

  15. Consumer Properties (4) • Querying data uses the standard SQL SELECT statement: rgma> SELECT * FROM <table_name> • Set the output format for results. 'table' formats the results in a table, tsv outputs tab-separated results and csv outputs comma-separated results: rgma> SET output table|tsv|csv output csv: 1,TestProducer,Start,10,Valeria,2006-12-07,22:07:36, set output tsv: 1 TestProducer Start 10 Valeria 2006-12-07 22:07:36

  16. rgma> set query latest Set query type to latest rgma> set maxage 1 minutes Set max age to 1 minutes rgma> set timeout 50 seconds Set timeout to 50 seconds rgma> set output tsv Set output format to 'tsv‘ rgma>select Application,Status FROM ITATutTable TestProducer Start TestProducer Step1 TestProducer Step2 Exercise 3: create a Consumer

  17. Continuos Producer and Consumer: (NOTE. Open 2 rgma client tool, one for Consumer the other for Producer.) • Consumer’s client: rgma> set query continuous rgma> set timeout 50 seconds rgma> set maxage 30 rgma> set output csv rgma> select * from ITATutTable rgma> • Producer’s client: rgma> set producer continuous rgma> insert INTO ITATutTable values(004,'TestProducer','Step3',30,'Valeria'); rgma> insert INTO ITATutTable values(005,'TestProducer','Step4',40,'Valeria'); rgma> insert INTO ITATutTable values(006,'TestProducer','Step5',50,'Valeria'); rgma> insert INTO ITATutTable values(007,'TestProducer','Step6',60,'Valeria'); rgma> write history Prod_comm.rgma Exercise 4: Producer & Consumer

  18. Continuos Producer and Consumer: (NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.) • Consumer’s client: rgma> set query continuous rgma> set timeout 60 seconds rgma> set output csv rgma> select * from ITATutTable • Producer’s shell: rgma -c “set producer continuous” rgma -c “insert INTO ITATutTable values(004,‘A','Step1',100,'Valeria');” rgma -c “insert INTO ITATutTable values(005,‘B','Step2',100,'Valeria');” Exercise 5: Producer & Consumer

  19. Continuos Producer and Consumer: (NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.) • Consumer’s client: rgma> set query continuous rgma> set timeout 60 seconds rgma> write results Results.rgma rgma> select * from ITATutTable • Producer’s shell: rgma -f Prod_comm.rgma Exercise 6: Producer & Consumer

  20. R-GMA Web Browser

  21. https://egris198.eela.ufrj.br:8443/R-GMA 3 1 2

  22. 3 https://egris198.eela.ufrj.br:8443/R-GMA 5 4 1 2

  23. 4 3 1 2

  24. Jobs using R-GMA commands

  25. #!/bin/sh echo "I'm doing application A...now I finished." $RGMA_HOME/bin/rgma <<EOF insert INTO ITATutTable values(001,'A','Step1',100,'Valeria'); EOF echo "I'm doing application B...now I finished." $RGMA_HOME/bin/rgma <<EOF insert INTO ITATutTable values(002,'B','Step2',100,'Valeria'); EOF echo "Bye bye" script.sh

  26. Type = "Job"; JobType = "Normal"; Executable=“/bin/sh"; Arguments = “script.sh"; StdOutput="stdout.log"; StdError="stderr.log"; InputSandbox={“script.sh"}; OutputSandbox={"stdout.log","stderr.log"}; my.JDL file

  27. >edg-job-list-match myJDL.jdl >edg-job-submit –o jobID myJDL.jdl >edg-job-status –i jobID >edg-job-get-output –i jobID Continuos Consumer Properties: >rgma rgma> set query continuous rgma> show query rgma> set timeout 3 minutes rgma> select * from ITATutTable Exercise 7

  28. R-GMA API to create Producer and Consumer

  29. User Producer and Consumer API available for Java, C, C++ and Python Users may by-pass API if they wish, but API is the easiest way to use R-GMA services

  30. Ingredients: • An application, called A, that does something and produce data; • A producer object used by A application to publish its relevant data; • A script to prepare the environment and launch the A application; • A JDL; • A rgma client or R-GMA Web Browser to act like a Consumer. Use Case for Monitoring Application

  31. Producer Properties Type: PRIMARY Storage type: MEMORY Termination Interval: 20 (minutes) Query type: CONTINUOUS Latest Retention Period: 60 (minutes) History Retention Period: 60 (minutes) . . . . . . . . . . ProducerProperties props = null; if (producerType.equals("CONTINUOUS")) { props = new ProducerProperties(Storage.MEMORY, 0); } else if (producerType.equals("LATEST")) { props = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST); } else if (producerType.equals("HISTORY")) { props = new ProducerProperties(Storage.DATABASE, ProducerProperties.HISTORY); } else { System.err.println("Invalid producer type (" + producerType + ")."); System.exit(1); } My Producer (Java): PP (1)

  32. . . . . . . . . . . PrimaryProducer pp = null; ResourceEndpoint endpoint = null; Try { ProducerFactory factory = new ProducerFactoryStub(); TimeInterval ti = new TimeInterval(20, Units.MINUTES); pp = factory.createPrimaryProducer(ti, props); String predicate = "WHERE Owner = '" + owner + "'"; TimeInterval historyRP = new TimeInterval(60, Units.MINUTES); TimeInterval latestRP = new TimeInterval(60, Units. MINUTES); pp.declareTable(tableName, predicate,historyRP,latestRP); ... My Producer (Java): PP (2)

  33. . . . . . . . . . . public void insert(String status, int percstat){ String insert = "INSERT INTO ITATutTable (COD_Test, Application, Status, PercStatus,Owner) VALUES (" + cod +",'" + appname +"','" + status + "'," + percstat + ",'" + owner + "')"; try { pp.insert(insert); } catch . . . . . . . . . . public void close() { try { pp.close(); } My Producer (Java)(3)

  34. public class A { public static void main(String[] args){ long startTime = System.currentTimeMillis() / 1000; if (args.length != 3) { System.err.println("Usage: java A <Cod_Test> <Owner> <producerType>"); System.exit(1); } PP mytest = new PP(new Integer(args[0]).intValue(),args[1],args[2]); for( int i = 0, j = 0; i <= 10; i++, j+=10) mytest.insert("Step"+i,j); mytest.close(); long endTime= System.currentTimeMillis() / 1000; System.out.println("resourceId=" + mytest.getResourceID() ); System.out.println("startTime=" + startTime); System.out.println("endTime=" + (System.currentTimeMillis() / 1000)); }} Application: A

  35. if [ -z "$RGMA_HOME" ] then export RGMA_HOME=$EDG_LOCATION fi export LD_LIBRARY_PATH=$RGMA_HOME/lib:$EDG_LOCATION/externals/lib #Java Environment if [ "$X509_USER_PROXY" != "" ] then securityProp="-DX509_USER_PROXY=$X509_USER_PROXY" else securityProp="-DTRUSTFILE=$TRUSTFILE" fi CLASSPATH="$CLASSPATH:$RGMA_HOME/share/java/glite-rgma-api-java.jar:$RGMA_HOME/share/java/glite-rgma-stubs-servlet-java.jar:$GLITE_LOCATION/share/java/glite-security-trustmanager.jar:$GLITE_LOCATION/share/java/glite-security-util-java.jar:$GLITE_LOCATION/externals/share/java/bcprov-jdk14-122.jar:$GLITE_LOCATION/externals/share/java/log4j.jar:." export CLASSPATH Script: startAPP.sh (1)

  36. COMM_EXE=A.class chmod +x $COMM_EXE # Arguments cod=$1 owner=$2 queryType="$3" if [ "$JAVA_HOME" != "" ] then COMM_JAVA=$JAVA_HOME/bin/java else COMM_JAVA=java fi $COMM_JAVA -classpath $CLASSPATH:. -DRGMA_HOME=$RGMA_HOME $securityProp A $cod $owner $queryType if [ $? -ne 0 ] then echo "Program $cod reported failure." 1>&2 exit 1 fi Script: startAPP.sh (2)

  37. Type = "Job"; JobType = "Normal"; Executable=“/bin/sh"; Arguments = "startAPP.sh 1001 Valeria CONTINUOUS"; StdOutput="stdout.log"; StdError="stderr.log"; InputSandbox={"startAPP.sh","pp.class“,”A.class”}; OutputSandbox={"stdout.log","stderr.log"}; JDL with User Producer Application

  38. R-GMA overview page. http://www.r-gma.org • R-GMA documentation in EGEE http://hepunx.rl.ac.uk/egee/jra1-uk/ • R-GMA in E-GRIS Site https://egris198.eela.ufrj.br/R-GMA • R-GMA API in E-GRIS Wiki Site More information

  39. Questions…

More Related