1 / 23

Multicriteria Driven Resource Management Strategies in GRMS

Multicriteria Driven Resource Management Strategies in GRMS. Krzysztof Kurowski, Jarek Nabrzyski, Ariel Oleksiak, Juliusz Pukacki Poznan Supercomputing and Networking Center, Poland. Resource management strategies. Grid Computing. GRMS Architecture. GRMS as a part of the GridLab.

shira
Download Presentation

Multicriteria Driven Resource Management Strategies in GRMS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multicriteria Driven Resource Management Strategies in GRMS Krzysztof Kurowski, Jarek Nabrzyski, Ariel Oleksiak, Juliusz Pukacki Poznan Supercomputing and Networking Center, Poland

  2. Resource management strategies Grid Computing The 4th Cracow Grid Workshop -CGW

  3. GRMS Architecture The 4th Cracow Grid Workshop -CGW

  4. GRMS as a part of the GridLab The 4th Cracow Grid Workshop -CGW

  5. GRMS as a part of the GridLab The 4th Cracow Grid Workshop -CGW

  6. Introduction to MC • Motivation • GRMS is a solution for non-dedicated resources • We consider a resource management problem as a multicriteriadecision making process with various stakeholders (end-users, administrators, etc.) • Different preferences of stakeholders (sets of criteria and their importance) • We need to aggregate somehow (negotiation and agreement processes are required) various criteria and stakeholders’ preferences • We focus on a compromise solution rather than the optimal one • We want to satisfy many stakeholders rather than the particular one • Various criteria and constraints depending on available information and scenarios • Requirements • Flexibility (in terms of criteria and multicriteria methods) • Re-use of common functionality (ranking, non-dominated solutions, solutions meeting constraints) The 4th Cracow Grid Workshop -CGW

  7. Introduction to MC • MCEvaluator • Implementation of needed multicriteria models and tools • Set of classes providing abstraction of entities used in multicriteria models • Job description/properties • Including user’s and administrator preferences • Taking into consideration multiple criteria, their importance, indifference thresholds and constraints • Two main efforts • Multi-criteria analysis engine – MCEvaluator • Including user’s preferences in a job description The 4th Cracow Grid Workshop -CGW

  8. MCEvaluator - Design The 4th Cracow Grid Workshop -CGW

  9. MCEvaluator - Model • Main entities • Criteria (objectives, soft constraints: user is interested in their greatest or lowest possible values) • Constraints (hard constraints: if is not satisfied solution is not taken into consideration) • Solutions (e.g. resources, schedules etc. along with description parameters) • MCEvaluator (decision point) • Multi-criteria methods • Evaluation function (e.g. weighted sum) • Non-dominated solutions • Lexicographic order • Rule-based systems • Other ... The 4th Cracow Grid Workshop -CGW

  10. Simple example End user 1 End user 2 Application 1 (e.g. Data analysis ) Application 2 (e.g. Data mining) Hard constraints (e.g. RSL… ClassAd, scripts) <Mem = 100MB>, <Storage = 1G> Memory Memory ??? ??? R1 R1 R2 R2 R3 R3 R4 R4 Storage Storage The 4th Cracow Grid Workshop -CGW

  11. Simple example End user 1 End user 2 Application 1 (e.g. Data analysis ) Application 2 (e.g. Data mining) MAXZ = 1*Mem + 2*Storage (where z is the objective function) MAXZ = 2*Mem + 1*Storage (where z is the objective function) Memory Memory R1 R1 R2 R2 R3 R3 R4 R4 Storage Storage The 4th Cracow Grid Workshop -CGW

  12. Job Description • Expressing user preferences • Criteria (I want as fast CPU as possible) • Weights (amount of memory is two times more important than a number of processors) • Constraints (amount of free memory must be greater than x) • Indifference thresholds (difference of 10MB is not significant) The 4th Cracow Grid Workshop -CGW

  13. Job Description- Schema <constraints><resConstraint parameter=„MEMORY"><constraintModel preferenceType="PRIORITY" optimizationType="GAIN"> <indiffThreshold>10</indiffThreshold> <value>1</value></constraintModel><constraintValue><min indiffThreshold="0">100</min><max indiffThreshold=„20">200</max></constraintValue> </resConstraint></constraints> The 4th Cracow Grid Workshop -CGW

  14. Examples • Selection of the best hosts for a job • Based on hosts parameters: CPU load, total memory, free memory, CPU count, CPU speed • Selection of the best host or the best queue • Based on estimated runtime and queue waiting time (+ estimated errors) taken from the prediction system • Selection of the best job to migrate (to release resources for a new job) • Based on parameters of the host after job migration and migration costs • Selection of the best queue in the local resource management system • Based on aggregated historical data taken from the GridLab Adaptive Service (Delphoi) The 4th Cracow Grid Workshop -CGW

  15. Selection of the best hosts for a job The 4th Cracow Grid Workshop -CGW

  16. Criteria CPU load CPU count CPU speed Total memory Free memory Constraints No constraints (resources filtered in the resource discovery phase) Solutions Set on the basis of parameters describing hosts //criteria NumericalCriterion[] criteria = new NumericalCriterion[5]; criteria[0] = new NumericalCriterion("cpuLoad", Criterion.GAIN); criteria[1] = new NumericalCriterion("cpuCount", Criterion.GAIN); criteria[2] = new NumericalCriterion("cpuSpeed", Criterion.GAIN); criteria[3] = new NumericalCriterion("memory", Criterion.GAIN); criteria[4] = new NumericalCriterion("memAvail", Criterion.GAIN); // //weights float[] weights = new float[5]; weights[0] = weightLoad; weights[1] = weightCPUCount; weights[2] = weightCPUSpeed; weights[3] = weightMemory; weights[4] = weightMemAvail; //evaluator mcEvaluators = new Vector(1); mcEvaluators.add(new MCFuncEvaluator("GRMSmatchmaking", criteria, weights, null)); Example 1 (Host Parameters) The 4th Cracow Grid Workshop -CGW

  17. Selection of the best host or the best queue The 4th Cracow Grid Workshop -CGW

  18. Criteria Time Error Standard Deviation Maximal – minimal time Constraints err <= stdev stdev < time/2 max-min <= time (+- 1 minute) Solutions Set on the basis of information from the prediction system //criteria NumericalCriterion[] criteriaMCT = new NumericalCriterion[4]; criteriaMCT[0] = new NumericalCriterion("time", Criterion.COST); criteriaMCT[0].setIndiffThreshold(1); criteriaMCT[1] = new NumericalCriterion("err", Criterion.COST); criteriaMCT[2] = new NumericalCriterion("stdev", Criterion.COST); criteriaMCT[3] = new NumericalCriterion("max-min", Criterion.COST); criteriaMCT[3].setIndiffThreshold(1); // //weights float[] weightsMCT = new float[4]; weightsMCT[0] = weightTime; weightsMCT[1] = weightErr; weightsMCT[2] = weightStdev; weightsMCT[3] = weightMaxMin; //constraints MetricConstraint[] constraintsMCT = new MetricConstraint[3]; //err <= stdev constraintsMCT[0] = new MetricConstraint("err", MetricConstraint.LE, null); constraintsMCT[0].setOtherMetric("stdev"); //stdev < time/2 constraintsMCT[1] = new MetricConstraint("stdev", MetricConstraint.LS, null); constraintsMCT[1].setOtherMetric("time"); constraintsMCT[1].setOtherMetricFactor(0.5f); //max-min <= time (+- 1) constraintsMCT[2] = new MetricConstraint("max-min", MetricConstraint.LE, null); constraintsMCT[2].setOtherMetric("time"); constraintsMCT[2].setIndiffThreshold(1); Example 2 (Time Prediction) The 4th Cracow Grid Workshop -CGW

  19. Selection of the best job to migrate (to release resources for a new job) The 4th Cracow Grid Workshop -CGW

  20. Criteria avgNumberWaitingJobs avgJobWaitingTime sdJobWaitingTime avgFreeNodes cpuTime maxJobs maxNodes Constraints cpuTime maxJobs maxNodes Solutions Set on the basis of information from the adaptive service // based on historic information (dynamic parameters)    criteria[0] = new NumericalCriterion("avgNumberWaitingJobs", Criterion.COST);    criteria[1] = new NumericalCriterion("avgJobWaitingTime", Criterion.COST);    criteria[2] = new NumericalCriterion("sdJobWaitingTime", Criterion.COST);    criteria[3] = new NumericalCriterion("avgFreeNodes", Criterion.GAIN);// based on queue configuration (static parameters)    criteria[4] = new NumericalCriterion("cpuTime", Criterion.GAIN);    criteria[5] = new NumericalCriterion("cpus", Criterion.GAIN);    criteria[6] = new NumericalCriterion("maxJobs", Criterion.GAIN);    criteria[7] = new NumericalCriterion("maxMem", Criterion.GAIN);    criteria[7] = new NumericalCriterion("maxNodes", Criterion.GAIN); Example 3 (Adaptive Service) The 4th Cracow Grid Workshop -CGW

  21. MCEvaluator – Next Steps • Development of the engine • Implementation of other multi-criteria methods • Negotiable constraints • Use in another scenarios • Use of other decision models (currently the evaluation function) • Scheduling with advance reservation • Resource co-allocations • Other ? The 4th Cracow Grid Workshop -CGW

  22. Conclusion • Flexible engine for criteria and constraintsmanagement – various: • criteria • constraints • decision making methods • Used in a few scenarios/testbeds • Currently multi-criteria evaluation function • Job description containing user preferences • Criteria (+ their importance) and constraints • It would be nice to use • Criteria in agreement negotiations • Criteria concerning deadlines, runtimes etc. The 4th Cracow Grid Workshop -CGW

  23. GRMS web page www.gridlab.org/grms/ Thank you! The 4th Cracow Grid Workshop -CGW

More Related