1 / 15

A generic approach to job tracking for distributed computing: the STAR approach

A generic approach to job tracking for distributed computing: the STAR approach. by V.Fine, J.Lauret, L.Hajdu. Introduction.

yitro
Download Presentation

A generic approach to job tracking for distributed computing: the STAR approach

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A generic approach to job tracking for distributed computing: the STAR approach by V.Fine, J.Lauret, L.Hajdu

  2. Introduction Job tracking, i.e. monitoring bundle of jobs or individual job behavior from submission to completion, is becoming very complicated in the heterogeneous Grid environment. This talk presents the principles of an integrating tracking solution based on components already deployed at STAR, none of which are experiment specific: a Generic logging layer and the STAR Unified Meta-Scheduler (SUMS). Job Tracking for Distributed Computing: the STAR Approach

  3. Logging overview • Most HENP experiment software includes a logging or tracing API allowing the display it a particular format of important feedback coming from the core application. • Typical so-called “HENP production” jobs initiate a chain of events where many components involved are often written in diverse languages, which do not offer a consistent and easily adaptable interface for logging important events Job Tracking for Distributed Computing: the STAR Approach

  4. STAR generic logging layer • The STAR Grid production environment contains a “generic logging layer”, built on top of a “logger” family derived from the “Jakarta log4j” project. That provides us the consistency across packages and frameworks. • “Jakarta log4j” is a community-supported project that includes the log4cxx, log4c, and log4perl packages (thereby supporting multiple computing languages) and encompassing four main component types: • loggers • appenders • layouts • filters • These four types are necessary to empower the developers with the capabilities to log messages according to message type and level, to control at runtime how these messages are formatted, and lastly to decide where they are reported. Job Tracking for Distributed Computing: the STAR Approach

  5. STAR generic logging layer (cont.) • STAR logger layer allows us to set independently the level of the verbosity for each STAR production “module”. This is convenient for the debugging of the code as a whole as modules and classes are uniquely identified by their message format. In addition, each developer can set the message output levels for their own modules. Job Tracking for Distributed Computing: the STAR Approach

  6. Generic logging layer and the STAR Unified Meta-Scheduler SUMS • SUMS is a "generic" gateway to user batch-mode analysis and allows the user to describe tasks based on an abstract job description language (SUMS’s architecture was designed around a module plug-and-play philosophy and is therefore not experiment specific). (More on SUMS: see talk: “Meta-configuration for dynamic resource brokering: the SUMS approach” ) Job Tracking for Distributed Computing: the STAR Approach

  7. SUMS job unique ID • The logger layer allows us utilizes a unique ID generated by SUMS for each task it handles and the set of jobs it creates; The ID is used for creating and updating Job records in the administrative database along with other vital job related information. Our approach does not require users to introduce any additional key to identify and associate the job with the database tables as the tree structure of information is handled automatically. Job Tracking for Distributed Computing: the STAR Approach

  8. Using the unique job id tag assigned by SUMS for the job tracking Job tracking SUMS assigns a unique tag for each job proxy log4perl Job_1 Job_2 Logger configuration log4cxx job_3 logger xml C++ log4j . . . GRID realm SUMS SUMS xml JAVA Job_N User’s meta-job File catalog Meta-log Job Tracking for Distributed Computing: the STAR Approach

  9. SUMS+log4j = powerful tracking • To exploit the SUMS + log4cxx capability one needs to make one step further by complementing the system with the dedicated logger and layout for each language the GRID infrastructure employs. • MySQLTrackingAppender – • establish a connection with the mysql Db (or its proxy) • execute the command provided and formated by the logger layout component. “Appender” has no idea what concrete command it is going to send to the MySQL server. Job Tracking for Distributed Computing: the STAR Approach

  10. SQL tracking “appender” + Tracking Layout = flexible Db interface • It is not mandatory to use MySQLTrackingAppender to direct the messages to MySQL Db. One can use th standard MySQLAppender from log4j kit. • It is not mandatory to this appender with JobTrackingLayout layout either. • But together they create a powerful tool to track the job submitted by SUMS Job Tracking for Distributed Computing: the STAR Approach

  11. JobTrackingLayout • Special layout, class derived from the “PatternLayout“ class • Doesn’t conflict with user’s will. The user is still free to format the message the way he/she thinks fit his/her current need. • On the other hands the SysAdmin, production manager, SUMS are allowed to add their own job logger configuration components to the job configuration file if any “JobTrackingLayout” can find and expand the $XXX like string to the $XXX run-time env. variable that can be present in the original user’s message to add the run-time env information to the MySQL command Job Tracking for Distributed Computing: the STAR Approach

  12. Mixing the program messages with the external parameters User’s code Logger<<“USER’S MESSAGE” DEBUG WANRING INFO FATAL Logger threshold SUMS attaches a unique tag for each real job Job log4j.xml appender custom filters selection patterns SUMS id Job environment Format rules layout Output message mySQLApprender consoleApprender . . . . . . . . Job Tracking for Distributed Computing: the STAR Approach

  13. XML configuration example: <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE configuration> <configuration xmlns='http://logging.apache.org/'> <appender name="stdout" class="org.apache.log4j.ConsoleAppender"> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%-3c{2}:%-5p - %m%n"/> </layout> </appender> <appender name="MYSQL" class="org.apache.log4j.MySQLTrackingAppender"> <layout class="org.apache.log4j.JobTrackingPatternLayout"> <param name="ConversionPattern" value="&quot;%-3c{2}&quot;)” > </layout> </appender> <root> <priority value ="FATAL" /> <appender-ref ref="stdout" /> </root> <logger name="QA"> <priority value ="DEBUG" /> <appender-ref ref="MYSQL" /> </logger> </configuration> Job Tracking for Distributed Computing: the STAR Approach

  14. MySQL job tracking tables relation Job Tracking for Distributed Computing: the STAR Approach

  15. Conclusion The propose approach of representing (sets of) GRID jobs submitted via SUMS system in a database does make easy to implement management, scheduling, and query operations as the user may list all previous jobs and get the details of status, time submitted, started, finished, etc… However to be able to track jobs through GRID realm the GRID components have to equipped with the logger layer STAR alike. Job Tracking for Distributed Computing: the STAR Approach

More Related