1 / 22

Accounting, Group Quotas, and User Priorities

Accounting, Group Quotas, and User Priorities . Why are you here?. To learn about How Condor chooses the next job to run How you can change which job will run How you can prioritize jobs by project instead of by user How you can group users into different projects

mariel
Download Presentation

Accounting, Group Quotas, and User Priorities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Accounting, Group Quotas, and User Priorities

  2. Why are you here? • To learn about • How Condor chooses the next job to run • How you can change which job will run • How you can prioritize jobs by project instead of by user • How you can group users into different projects • How you can assign usage minimums to groups of users

  3. What job runs next? • A condor “queue” is not FIFO! • Determined by a balancing the wants & needs of three entities • The user (schedd) • The pool administrator (negotiator) • The machine owner (startd) • All comes together in the negotiation cycle

  4. negotiator collector startd startd schedd schedd startd 1. Startds send machine ads 2. Schedds send submittor ads

  5. Machine Ads MachineMyType = "Machine" TargetType = "Job" Name = "vm6@romano.cs.wisc.edu" Machine = "romano.cs.wisc.edu“ Requirements = LoadAvg < 0.5 Rank = 0.0 Disk = 2019048 LoadAvg = 0.000000 KeyboardIdle = 1018497 Memory = 512 Cpus = 1 Mips = 124122 condor_status –l romano.cs.wisc.edu

  6. Submittor Ads MyType = "Submitter" TargetType = "" Machine = "rosalind.cs.wisc.edu" ScheddIpAddr = "<128.105.166.39:42190>" Name = “matthew@cs.wisc.edu” RunningJobs = 1 IdleJobs = 1 HeldJobs = 0 MaxJobsRunning = 500 StartSchedulerUniverse = TRUE MonitorSelfImageSize = 9164.000000 MonitorSelfResidentSetSize = 2432 condor_status -sub matthew@wisc.edu -l

  7. negotiator collector startd Let’s look inside the negotiator during a negotiation cycle… startd schedd schedd startd 1. Startds send machine ads 2. Schedds send submittor ads

  8. Inside the Negotiator…

  9. negotiator accountant

  10. Negotiation Cycle • Schedds send requests one at a time, sorted by job priority • Get all startd and submittor ads • Get user priorities for all submitters, or accounting principles. (via Name attribute in submitter ad) • Sort submitter ads • Talk to schedds in accounting principle order

  11. For Each Job: • Find all machine ads that match • Sort machine ads that match by:  NEGOTIATOR_PRE_JOB_RANK  Job Ad’sRANK  NEGOTIATOR_POST_JOB_RANK • Is the machine ad candidate already running a job? Priority preemption if PREEMPTION_REQUIREMENTS evaluates to True. • Give the schedd the match, or tell it no match found. Schedd responds w/ next request (maybe skipping to the current AutoCluster).

  12. Some observations • Job priority (condor_prio) will not allow one user to run ahead of another user. • Job priority is specific per user per schedd.

  13. Examples Job says : Rank = Memory Config file does not define: NEGOTIATOR_PRE_JOB_RANK User will then ALWAYS get the highest memory machine, even if already being used by a lower priority user.

  14. Examples Job says : Rank = Memory Config file says: NEGOTIATOR_PRE_JOB_RANK = RemoteUser =?= UNDEFINED User will then get the highest IDLE memory machine, and will only preempt a user if there are no idle machines match.

  15. Accounting Groups I don’t care about WHO submitted the job. How do I change the accounting principle?

  16. Account Groups, cont • In job submit file executable = foo universe = vanilla +AccountingGroup = “Project44” queue

  17. A given group should have priority on 50 nodes of my 500 machine cluster. How? Answer A: Startd Rank Answer B: Group Quotas

  18. Group Quotas – Config Params GROUP_NAMES - list the recognized group names.Example:      GROUP_NAMES = group-cms, group-infn GROUP_QUOTA_<groupname> - the number of machines 'owned' by this group.    Example:      GROUP_QUOTA_group-cms = 10      GROUP_QUOTA_group-cms = 5 GROUP_AUTOREGROUP - set this to either True or False.  Defaults to     false.  If true, then users who submitted to a specific group     will also negotiate a second time with the "none" group, allowing     group jobs to be matched w/ idle machines even if the group is     overquota.

  19. Negotiation w/ Group Quotas • Matchmaker first negotiates for groups, sorted by how far they are under quota. • Negotiation within a group follows the exact same algorithm as before. • THEN, negotiate for all users that are not in a group as before.

  20. Lots o “tools” to get what you want • USER: Job Requirements, Job Rank • ADMIN: user priorities, accounting groups, accounting group quotas, preemption_requirements, negotiator_pre|post_job_rank. • OWNER: Machine Requirements, Machine Rank.

  21. Challenge Question I want LOW, MED, and HIGH strict priority job types. But at each priority level, I want fair share. HOW??

  22. Questions?

More Related