1 / 16

STAR Scheduling status

This project aims to stabilize the user interface for job submission and provide an architecture that allows for easy changes. It includes components such as Job Initializer, Policy, File Catalog integration, Dispatcher, and other functionalities.

leonc
Download Presentation

STAR Scheduling status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. STAR Scheduling status Gabriele Carcassi 9 September 2002

  2. Objectives • Have something for September • Stabilize the user interface used to submit jobs, based on the user perspective • Provide an architecture that allow easy change • Provide a way for the administrator to change the behavior of the system

  3. JobInitializer Policy Perl interface Dispatcher MySQL LSF STAR Scheduling architecture Scheduler / Resource broker (?) UI UJDL File Catalog Current architecture for job submission Queue manager

  4. User interface • Driven by use cases, and not by the tools used to implement it • user basically gives the job and the list of input files, which can also be a catalog query • User specify what he wants to do, and not how to do it • simpler to use • gives the administrator more flexibility in the implementation

  5. User interface • User job description in XML • Scheduler developed at Wayne State uses XML • Easy to extend: • ex. multiple ways to describe the input <input URL=“...” /> <input filename=“...” /> • Parsers already available

  6. Job Initializer • Parses the xml job request • Checks the request to see if it is valid • Checks for elements outside specification (typically errors) • Checks for consistency (existence of input files on disk, ...) • Checks for requirements (require the output file, ...) • Creates the Java objects representing the request (JobRequest)

  7. Job Initializer • Current implementation • Strict parser: any keyword outside the specification stops the process • Checks for the existence of the stdin file and the stdout directory • Forces the stdout to prevent side effects (such as LSF would accidentally send the output by mail)

  8. Policy • From one request, creates a series of processes to fulfill that request • Processes are created according to farm administrator’s decisions • The policy may query the file catalog, the queues or other middleware to make an optimal decision

  9. Policy • We anticipate a lot of the work in finding an optimal policy • Policy is easily changeable, to allow the administrator to change the behavior of the system

  10. Policy • Current policy • The query is resolved by simply querying the catalog • Divide the job into several processes, according to where the input file is located • No more than 10 input files per job

  11. File Catalog integration • In the job description a user can specify one or more queries • Depending on how these queries are resolved, the farm can be more or less efficient • Mechanism to execute the query is separate from the query description • easy to change catalog implementation

  12. File Catalog integration • Current implementation: • Very simple to allow fast implementation • Forwards the query as it is to the perl script interface of STAR catalog • main advantage: same syntax for the user • No “smart” selection is made • no effort is done for selecting those files that would optimize the use of the farm

  13. Dispatcher • Talks to the underlying queue system • Takes care of creating the script that will be executed • Creates environment variables and the file list

  14. Dispatcher • Current implementation: • creates file list and script in the directory where the job was submitted from • creates environment variables containing the job id, the list of files and all the files in the list. • creates a command line for LSF • submits job to LSF

  15. Other functionalities • Log • The logging services provided in Java 1.4 are used to create a detailed log • Each entry has a level attribute (FINEST, FINER, FINE, INFO, WARNING, SEVERE), and the log can be selected to produce output only starting from one level • We will use FINEST during beta, INFO during the first months of production, and WARNING after that • Logging goes on behind the back of the user, providing full information about usage essential to trace bugs and problems associated with the policy.

  16. Conclusion • The tool is available and working • beta quality: works reliably, some small feature might be needed, QA test still required. • Allows the use of local disks • Architecture is open to allow changes • Catalog implementation (MAGDA, RLS, GDMP, ... ?) • Dispatcher implementation (Condor, Condor-g – Globus, ... )

More Related