Batch Systems P. Nilsson, PROOF Meeting, October 18, 2005

1. Batch Systems P. Nilsson, PROOF Meeting, October 18, 2005

2. Overview • Sun’s Grid Engine [Open Source] • What is the Grid Engine project? • Daemon components • Execution daemon • Priorities • Job life cycle • Platform LSF [Commercial system] • What is Platform LSF? CERN Batch Service • Product features • Architecture • Load sharing • Job life cycle • Other Batch Systems • OpenPBS, Condor, BQS, … • Maui Scheduler [Open Source] Paul Nilsson

3. Sun’s Grid Engine What is the Grid Engine project? • “An open source community effort to facilitate the adoption of distributed computing solutions”, sponsored by Sun • The project provides distributed resource management software for wide ranging requirements from compute farms to grid computing The Grid Engine has been ported to many operating systems, including Sun Solaris, Linux, SGI IRIX, Compaq/HP Tru64, IBM AIX, HP HP/UX, Apple Mac OS/X and others. The project welcomes those who are interested in implementing new ports or in taking over the maintenance of an existing port Good documentation… More information at http://gridengine.sunsource.net/ Paul Nilsson

4. Sun’s Grid Engine Grid Engine Components • Qmaster [“queue master”]controls the overall behavior in a cluster, responsible for answering requests from clients and for delivering dispatched jobs to the assigned Execd's • Schedd [“scheduling daemon”] gets notified about all scheduling relevant information. Resulting scheduling decisions are sent as orders to Qmaster • Execd [“execution daemon”]provides Qmaster with information about utilization and availability of resources. A job sent to Execd is started by writing all relevant information into files describing the job and forking a “Shepherd”. After Shepherd’s termination, Execd reports back to Qmaster • Shepherd starts all kinds of jobs according to what he finds in the per-job configuration files written by Execd • Commd [“communication daemon”]handles network communication in a cluster • Shadowd [“shadow daemon”]detects failures of the Qmaster and starts a new Qmaster if necessary Paul Nilsson

5. Sun’s Grid Engine Execd – The Execution Daemon The Execution Daemon is the instance that: • Starts jobs • Controls jobs (e.g. it can suspend/unsuspend a job, reprioritize the processes associated with a job, etc) • Gathers information about jobs (e.g. resource usage, exit code, etc) • Gathers information about the execution host it controls (e.g. load, free memory, etc) There is one execd on each host of a cluster Paul Nilsson

6. Sun’s Grid Engine On Priorities… The Grid Engine has the feature of a share-based scheduler, where each job gets a certain share of the system resources • The sum of all shares for a job is expressed in tickets. A job has a certain number of tickets enabling it to run with certain process priorities • If multiple jobs are running concurrently on a host, their different share of system resources (their different number of tickets) can be mapped to priorities in the OS • Setting priorities in the OS is done by either setting the nice value for all processes of a job or by using special priority mapping facilities provided by the OS • The Grid Engine reassigns the number of tickets per job in a regular interval. It then maps the number of tickets of a job to nice values (or another operating system priority representation) and renices all processes of the job Much more information about the scheduler is in the documentation… Paul Nilsson

7. Sun’s Grid Engine Job Life Cycle • Execds report load information to Qmaster • User submits job using qsub command • Qmaster notifies Schedd about new job • Schedd dispatches job to an Execd • Qmaster delivers job to Execd; Execd starts job using Shepherd • At job end Execd notifies Qmaster about job finish • Qmaster feeds jobs resource consumption in accounting database Execd1 Execd2 ExecdN 1 1 1 5 4 6 Schedd Qmaster 3 7 2 qsub Paul Nilsson

8. Platform LSF What is Platform LSF? • “Platform LSF (Load Sharing Facility) is a [commercial] workload management solution that optimizes the use of enterprise-wide resources by providing transparent, on demand access to valuable computing resources” CERN Batch Service • CERN Batch Service provides an “LSF” farm with 1500 dual-processor machines for data analysis and simulation (used CPU time is accounted to the experiments!!!) • LXPLUS is used for the public logon (i.e. for the job submits) • Depending on the load and resource requirements of jobs up to 3 jobs are running in parallel on the same node More information at http://batch.web.cern.ch/batch/ http://www.platform.com/Products/Platform.LSF.Family/Platform.LSF http://www.hp.com/techservers/software/lsf.html Paul Nilsson

9. Platform LSF Product features • Dynamic Load Balancing by continuously monitoring of system resources; CPU and memory usage, swap space, software license availability (!) • Resource-based Queuing and Scheduling. Resources are dynamically managed based on policies, schedules and thresholds; jobs submitted to any network-based queues are automatically processed as resources become available • Optimal Resource Sharing. Continuous resource management, even in the event of host failures: failed jobs are automatically re-run and failed servers restarted • Administrative Control and Policies. Admins can suspend, stop, and submit jobs from any node in a network; users can modify their own jobs once submitted to queues. Varied options are available for configuring workload policies, supporting resource sharing by users, user groups, and projects Paul Nilsson

10. Platform LSF Load Sharing To achieve load sharing, LSF must have up to date information about the load on each machine in a cluster. The Load Information Manager (LIM) component is responsible for this. A LIM daemon runs on each host of the cluster. It gathers information about its host and makes the information available to all hosts. The information is organized as a load vector. The load vector comprises a number of load indices as described in the following table Load index Description r15s Load average for last 15 seconds r1m Load average for last minute exponentially averaged CPU run queue lengths r15m Load average for last 15 minutes ut Percent CPU utilization averaged over last minute pg Paging (in/out) activity over last 20 seconds ls Number of login sessions it Idle time - number of minutes since last keyboard or mouse activity tmp Available space (MB) in /tmp file system swp Available swap space (MB) mem Available real memory (MB) Paul Nilsson

11. Platform LSF Job Life Cycle • User submits a job to LSF for execution • The submitted job proceeds through the batch library to the Load Information Manager (LIM) • LIM communicates the job's information to the cluster's master LIM. Periodically, the LIM on individual machines gathers its 12 built-in load indices and forwards this information to the master LIM [see previous slide] • The master LIM determines the best host to run the job and sends this information back to the submission host's LIM • (Information about the chosen execution host is passed through the batch library) • Information about the host to execute the job is passed back to the bsub process or lsb_submit() function • To enter the batch system, bsub or lsb_submit() sends the job to the batch library • Using batch library services, the job is sent to the mbatchd running on the cluster's master host 1 Paul Nilsson

12. Platform LSF Job Life Cycle • The mbatchd puts the job in an appropriate queue and waits for the appropriate time to dispatch the job. User jobs are held in batch queues by mbatchd, which checks the load information on all candidate hosts periodically • The mbatchd dispatches the job when an execution host with the necessary resources becomes available where it is received by the host's sbatchd • sbatchd controls the execution of the job and reports the job's status to mbatchd. The sbatchd creates a child sbatchd to handle job execution • The child sbatchd sends the job to the Remote Execution Server (RES) • The RES creates the execution environment to run the job • The job is run in the execution environment • The results of the job are sent to the email system • The email system sends the job's results to the user 2 Paul Nilsson

13. Other Batch Systems What other batch systems are on the market? • OpenPBS- Open source Portable Batch System [Unsupported version – development stopped in 1999]. “Flexible batch queuing system developed for NASA in the early to mid-1990s. It operates on networked, multi-platform UNIX environments”. Developed into a commercial PBS Pro version (http://www.openpbs.org, compilation requires hacker intervention..). Public home: http://www-unix.mcs.anl.gov/openpbs • Condor – “Specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queuing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. Users submit their serial or parallel jobs to Condor, Condor places them into a queue, chooses when and where to run the jobs based upon a policy, carefully monitors their progress, and ultimately informs the user upon completion” (http://www.cs.wisc.edu/condor) • BQS - Batch Queuing System. Some information at CC-IN2P3 web site (http://webcc.in2p3.fr/man/bqs/intro) • … Paul Nilsson

14. Maui Scheduler Open source batch queuing and scheduling software designed to schedule parallel jobs • Maui can schedule the order of job execution for queued jobs (from other batch systems) • Has lots of scheduling concepts; FIFO (First-in first-out) like reservations, back-filling of jobs, job priorities, time-of-day scheduling, etc • Written in Java • Maui Scheduler has been designed to communicate directly with a database through an abstraction layer BUT currently MySQL is the only one implemented [MySQL is required] • Considered for CASTOR2 (until LSF was chosen) What is “backfilling”? • Maui allows a lower priority job to be executed before a higher priority job if it does not delay the start of the prioritized job [apparently not found in other schedulers] More information at http://mauischeduler.sourceforge.net Paul Nilsson