1 / 43

Faucets Tutorial

Faucets Tutorial. Presented by Esteban Pauli and Greg Koenig Parallel Programming Lab, UIUC. Outline. System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy. Outline. System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy.

selena
Download Presentation

Faucets Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Faucets Tutorial Presented by Esteban Pauli and Greg Koenig Parallel Programming Lab, UIUC Charm++ Workshop, 2005

  2. Outline • System Overview • Cluster Scheduler • Meta Scheduler • Writing a Scheduling Strategy Charm++ Workshop, 2005

  3. Outline • System Overview • Cluster Scheduler • Meta Scheduler • Writing a Scheduling Strategy Charm++ Workshop, 2005

  4. Where should I submit my job? Cluster Cluster Cluster Cluster User User User User Current Situation Charm++ Workshop, 2005

  5. I’ll submit my job to Faucets!!! Cluster Cluster Cluster Faucets System User User User Faucets System Charm++ Workshop, 2005

  6. Cluster Cluster Cluster Faucets System User Faucets System 1. User submits job to Faucets system Job requirements Charm++ Workshop, 2005

  7. Cluster Cluster Cluster Faucets System User Faucets System 2. Faucets forwards requests to clusters meeting minimum requirements Job requirements Job requirements Charm++ Workshop, 2005

  8. Cluster Cluster Cluster Faucets System User Faucets System 3. Clusters analyze job specs and return bids Bid Bid Bids Charm++ Workshop, 2005

  9. Cluster Cluster Cluster Faucets System User Faucets System 4. User selects winner Winner selected Winner selected Charm++ Workshop, 2005

  10. Cluster Cluster Cluster Faucets System User Faucets System 5. Winner runs job, user monitors progress Charm++ Workshop, 2005

  11. Outline • System Overview • Cluster Scheduler • Meta Scheduler • Writing a Scheduling Strategy Charm++ Workshop, 2005

  12. Central Server Cluster Daemon Scheduler Database Cluster User User System Architecture Charm++ Workshop, 2005

  13. Why a New Scheduler? • Current schedulers try to maximize throughput – good for showing cluster is busy, but can be bad for users • Users worry about deadlines, priorities, fairness, etc. • Profit centers worry about profits • Need good interface between cluster scheduler and meta scheduler • Want scheduler that can leverage run-time system (Charm++ checkpoint/restart, shrink/expand) Charm++ Workshop, 2005

  14. Request Server Job Monitor Cluster Database Cluster Monitor Scheduler User Job scheduler Strategy Scheduler Design Charm++ Workshop, 2005

  15. Installation • Install MySQL database • Install Charm++, MPI • ./configure; (edit config file); make; make install • sh sqlwriter.sh | mysql –user=root –p • Set up node list • ./startScheduler • Done!!! Charm++ Workshop, 2005

  16. Job Submission • ufrun (uni-processor), frun (Charm++), frun-mpi (mpi) used for interactive jobs • ufsub, fsub, fsub-mpi used for batch jobs • Options: +n, +p, +ppn, -stdout, … • See manual for full details Charm++ Workshop, 2005

  17. Job Control • Job monitoring: fjobs • Options: -user <name>, -full, id • Sample output: UserId JobId # Nodes Head Node Status Name Time Remaining ------ ----- ------- --------- ------ ---- -------------- fuser 63503 1 arch017.c RUNNING my_exe 0:06:39:28 • Job deletion: fkill • Options: -u <name>, id Charm++ Workshop, 2005

  18. Outline • System Overview • Cluster Scheduler • Meta Scheduler • Writing a Scheduling Strategy Charm++ Workshop, 2005

  19. Disclaimer • Current code base developed as a proof of concept • Code is not yet production quality • Code works, but has not been tested thoroughly • Code has some security issues • Use at your own risk, and please report bugs • Code will be updated within the next year Charm++ Workshop, 2005

  20. Central Server Cluster Daemon Scheduler Database Cluster User User System Architecture Charm++ Workshop, 2005

  21. Central Server • Responsible for keeping all information about users and clusters in system • Responsible for forwarding users’ job requests to clusters • Responsible for dispute arbitration • Responsible for keeping account balances Charm++ Workshop, 2005

  22. Central Server: Database • Cluster table: contains information about confederated clusters mysql> create table Cluster ( domainName text not null, port int not null, status text not null, acctId int not null ); Charm++ Workshop, 2005

  23. Central Server: Database (cont.) • User table: contains information about registered users mysql> create table users ( userid text not null, password text not null, localCluster text not null, acctId int not null ); Charm++ Workshop, 2005

  24. Central Server: Database (cont.) • Account table: keeps account balances for clusters and users mysql> create table accounts ( clusterid text not null, acctId int not null, balance int not null, pbalance int not null); Charm++ Workshop, 2005

  25. Central Server: Database (cont.) • Job table: keeps track of all running and completed jobs mysql> create table Jobs ( JobID text not null, User text not null, Status text not null, ClusterID text not null); Charm++ Workshop, 2005

  26. Central Server: Installation • Compilation & Configuration: • cd faucets; make • Edit faucets/cs/db.properties • Get and install JDBC • Running the central server: • cd faucets • java -cp .:/path/to/mm.mysql-2.0.8-bin.jar TheServer • As users and clusters join, update DB Charm++ Workshop, 2005

  27. Central Server Cluster Daemon Scheduler Database Cluster User User System Architecture Charm++ Workshop, 2005

  28. Cluster Daemon • Purpose: provide interface between central server and cluster scheduler • No user intervention • Installation: cd faucets; make • Usage: • cd faucets • java -cp .:./common/TB.jar cd.ClusterDaemon <CS_hostname> <CS_port> /tmp/ Charm++ Workshop, 2005

  29. Central Server Cluster Daemon Scheduler Database Cluster User User System Architecture Charm++ Workshop, 2005

  30. Command Client • Installation: cd faucets/cc; make • Some common commands • Job submission java cc.FaucetCLI <central_server_DNS> <port> <application name> [-input file1,file2,...,filen] [<args>] • Retrieving output files java FaucetCLI GetFile <job-id> <file-name> Charm++ Workshop, 2005

  31. Faucets GUI Charm++ Workshop, 2005

  32. Central Server Cluster Daemon Scheduler Database Cluster User User System Architecture Charm++ Workshop, 2005

  33. Outline • System Overview • Cluster Scheduler • Meta Scheduler • Writing a Scheduling Strategy Charm++ Workshop, 2005

  34. Faucets Scheduling Framework • In many ways Faucets can be thought of simply as a framework for creating cluster scheduling solutions • Any Faucets deployment has some scheduling objective that it tries to achieve • Traditional FIFO scheduling • On-demand scheduling – driven by workloads and the priorities that users have to access resources • Resource bartering – driven on an economic basis (“where can this job be run for the least cost?”) Charm++ Workshop, 2005

  35. Schedule Strategies • The scheduling method used by Faucets can readily be changed by writing a scheduling strategy • The scheduling strategy can do interesting things that take advantage of features in lower level runtime systems (e.g., Charm++ Adaptive Jobs that shrink/expand to better utilize cluster) • Scheduling strategy code is a C++ class • Implement to reflect the scheduling method • Recompile the cluster scheduler executable Charm++ Workshop, 2005

  36. Schedule Strategy Examples • PriorityFIFOStrategy • Jobs have assigned priorities • Jobs of a given priority are scheduled FIFO • Jobs of a higher priority can preempt jobs of a lower priority • LimitFIFOStrategy • Jobs are scheduled FIFO • The number of short/long jobs that a given user may be running simultaneously is limited to prevent resource domination • GanttChartStrategy • Jobs are scheduled by arranging them on a Gantt Chart • Predictions can be made about whether a given job can be completed before a user-specified deadline Charm++ Workshop, 2005

  37. Implementing a Strategy (1) • To implement a new strategy • Inherit from SchedulingStrategy base class • Implement four methods class PriorityFIFOStrategy : public SchedulingStrategy { public: PriorityFIFOStrategy (int nodes); float is_available (Job *job, Job *wait_queue, Job *run_queue); void allocate_nodes (Job *wait_queue, Job *run_queue); void addjob (char *username, int num_procs); void removejob (char *username, int num_procs); }; Charm++ Workshop, 2005

  38. Implementing a Strategy (2) float is_available (Job *job, Job *wait_queue, Job *run_queue); • Method parameters • job – a pointer to an incoming job • wait_queue – a pointer to the queue of all waiting jobs • run_queue – a pointer to the queue of all running jobs • Method returns a float • 0.0 to 1.0 – the utilization of the cluster if the incoming job is accepted for execution (0% to 100%) • -1.0 - the incoming job cannot be accepted for immediate execution • This method is used (indirectly) by the Cluster Daemon to make the choice of target cluster when doing metascheduling Charm++ Workshop, 2005

  39. Implementing a Strategy (3) void allocate_nodes (Job *wait_queue, Job *run_queue); • Method parameters • wait_queue – a pointer to the queue of all waiting jobs • run_queue – a pointer to the queue of all running jobs • Method examines each Job object in the wait_queue and updates it by allocating available cluster nodes to the job so the scheduler can launch it • For Charm++ jobs, normally try to allocate max_nodes first and then min_nodes after that to fulfill the request Charm++ Workshop, 2005

  40. Implementing a Strategy (4) void addjob (char *username, int num_procs); void removejob (char *username, int num_procs); • Method parameters • username – the user submitting a new job into the cluster • num_procs – the number of processors the new job is allocated • Method can be used to enforce some characteristic about the number of processors allocated to any given user (e.g., limit the total number of processors that any given user is allocated) Charm++ Workshop, 2005

  41. Instantiating a Strategy • The scheduling strategy is instantiated in the Scheduler class constructor located in Scheduler.C strategy = new PriorityFIFOStrategy (num_nodes); • The total number of nodes in the cluster is provided to the strategy’s constructor • That’s it! The scheduler code calls into your custom Strategy class whenever it needs to make a decision about whether a new job can be scheduled, to allocate nodes to the job, etc. Charm++ Workshop, 2005

  42. Questions? Charm++ Workshop, 2005

  43. Thanks Charm++ Workshop, 2005

More Related