1 / 22

GridPP: Executive Summary

GridPP: Executive Summary. Tony Doyle. Exec 2 Summary.

walterbrown
Download Presentation

GridPP: Executive Summary

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GridPP: Executive Summary Tony Doyle Oversight Committee

  2. Exec2 Summary • The GridPP Project is now underway and has met 21% of its original targets with 86% of the metrics within specification. This summary highlights various issues with respect to resource monitoring and project management, where the status of GridPP2 planning is described in the context of the “get fit” plan requested by the Oversight Committee. gLite 1 was released in April as planned but components have not yet been deployed or their robustness tested by the experiments. Service Challenge (SC) 2 addressing networking was a success at CERN and the Tier-1 and SC3 addressing file transfers for the experiments is about to commence. Hardware levels at the Tier-1 in 2007-08 and the deployment of Tier-2 resources are current concerns although these are mitigated by the present under-utilisation of resources. Oversight Committee

  3. RAL joins labs worldwide in successful Service Challenge 2 • The GridPP team at Rutherford Appleton Laboratory (RAL) in Oxfordshire recently joined computing centres around the world in a networking challenge that saw RAL transfer 60 terabytes of data over a ten-day period. A home user with a 512 kilobit per second broadband connection would be waiting 30 years to complete a download of the same size. Oversight Committee

  4. gLite 1 Oversight Committee

  5. 100 green sites sitting on a grid • Thu 16 Jun 2005 • Last week the UK CIC-on-duty team celebrated the milestone of having 100 sites passing the Sites Functional Test. Thanks to all the sites who acted promptly to trouble tickets raised by the UK team during their shift. Oversight Committee

  6. Current concern 1. under-utilisation • Under -utilisation of existing Tier-1/A resources • improving overall and w.r.t. Grid fraction from 2004 to 2005 Non-Grid Grid Oversight Committee

  7. Current concern2. under-delivery • The current situation is somewhat better than these 2005 Q1 numbers indicate • Some late procurements (OK given under-utilisation) • Technical problems (being overcome) Oversight Committee

  8. GridPP Deployment Status 30/6/05 (9/1/05) Measurable Improvements Oversight Committee

  9. Actions GridPP to submit the proposal for LCG phase 2 funding to the Committee prior to its submission to Science Committee (minute 4.9). • Done. 27 page report inc. input from OC at http://www.gridpp.ac.uk/docs/gridpp2/SC_GridPP2_LCG_1.0.docunfunded GridPP to clarify the situation with regard to ATLAS production run tests for the next physics workshop (minute 5.3). • See News Item http://www.gridpp.ac.uk/news/-1119651840.463358.wlg • (and slide) GridPP to provide an update on progress resolving problems caused by mismatches between local batch systems and the capabilities of the grid Resource broker (minute 6.3). • (See slide) GridPP to more fully document its alignment with each of the individual experiments (minute 15.2). • An experiment engagement questionnaire has been used (initial input in February and further [updated] input in June). See http://www.gridpp.ac.uk/eb/workdoc/gridusebyexpts_0605.doc Oversight Committee

  10. ATLAS steps up Grid production Oversight Committee

  11. RB Action GridPP to provide an update on progress resolving problems caused by mismatches between local batch systems and the capabilities of the grid Resource broker (minute 6.3). • The problem of connecting the local CE to a batch queue is largely overcome – many (all shared) sites now do this. • There were problems subsequently deploying the accounting system (APEL) to point to the local batch system. • Overcome (13 ex 18 sites), but not as straightforward as it could be. • The JDL from the job is not passed to the local system. Hence there is no way for the local scheduler to use info from the Grid scheduler. • This is a limitation from a (shared) site viewpoint (attempting to balance Grid and local jobs). • The short term solution is to set up separate batch queues. • It is not a limitation for the experiments (affects efficiency). • It is noted as a requirement and it is intended that this will be delivered in Year 2 of JRA1 for the WMS. Oversight Committee

  12. Actions GridPP to define its usage policy with respect to Tier-1 allocations(minute 15.4). • See http://www.gridpp.ac.uk/docs/oversight/GridPP-PMB-57-Tier1A_1.0.doc and documents within (“fair shares” using PPARC Form X information) GridPP to produce an updated risk register (minute 15.5). • Incorporated in the new Project Map at (with 7 “high” risks) http://www.gridpp.ac.uk/pmb/ProjectManagement/GridPP2_ProjectMap_2.htm GridPP to produce a “get-fit” plan for production metrics (minute 15.6). • See Metrics and Deployment document http://www.gridpp.ac.uk/docs/oversight/GridPP-PMB-64-Metrics.doc and its incorporation into the Project Map GridPP to define its metrics for job success (minute 15.7). • Adopted EGEE-wide definition at http://ccjra2.in2p3.fr/EGEE-JRA2/QAmeasurement/showstatsVO.php (See slides) GridPP to produce a statement of intent regarding its adoption of gLite (minute 15.8). • See Middleware Selection document http://www.gridpp.ac.uk/docs/oversight/GridPP-PMB-65-Middleware.doc Oversight Committee

  13. Metrics Action GridPP to define its metrics for job success (minute 15.7). • GridPP adopts the EGEE-wide definition at http://ccjra2.in2p3.fr/EGEE-JRA2/QAmeasurement/showstatsVO.php The (web-based) QA system accounts for Workload Management System registered job successes (that can then be categorised by Virtual Organisation or Resource Broker) Before introducing the figures it should be understood that there are caveats: • It only measures what the WMS “sees” • doesn't catch failure of WMS to register job in the first place (but this is a rare occurrence) • if a job half way through the script fails (for example tries but fails to copy a file) but the script completes successfully then WMS sees everything as OK. • If a VO (e.g. LHCb) deploys an agent then the WMS only registers the success of the initial (python) script: strategy enables higher overall LHCb performance (combined push-PULL model). (This currently leads to other problems in overall accounting should contention become an issue). • Overall: an end user may see either: • 1. a worse efficiency • failed job for other hidden e.g. data management problems • 2. a better efficiency by • choosing selected sites according to the Site Functional Test performance index; • deploying an agent to initiate real jobs at sites where the agent succeeded. • Physicists are “smart” and now “see” > 90% efficiency but the definition here is one defined within a given VO adopting their own methods (and from informed input from people currently submitting jobs to the system). Oversight Committee

  14. Overview Integrated over all VOs and RBs for first half of 2005 Successes/Day 13806 Success %64% • Key point: Improving from 42% to 78% during 2005 [For the UK RB (lcgrb01.gridpp.rl.ac.uk) Successes/Day 319 Success %69%] Oversight Committee

  15. LHC VOs ALICE ATLAS CMS LHCb Successes/Day N/A 2796 452 3463 Success %42% 83% 61% 68% Oversight Committee

  16. Other VOs BaBar CDF D0 BioMed Successes/Day 37 1 207 1074 Success %76% 30% 84% 76% Oversight Committee

  17. The “Get Fit” Plan • Set SMART (Specific Measurable Achievable Realistic Time-phased) Goals Oversight Committee

  18. “I take it plea bargaining is out of the question?” Oversight Committee

  19. Our problems… • 0.104: Number of LCG/EGEE job slots published by the UK. The current total is 2477 and the target was 3000. • 0.106: GridPP KSI2K available: By the end of March 2005 the combined Tier-1 and Tier-2 CPU power was expected to be 5184 KSI2K compared to 2277 KSI2K achieved. This number is dominated by the 4397 KSI2K expected from the Tier-2s which has been slowly becoming available. • 0.108: GridPP disk storage available: Similar to 0.106 above. Only 280TB available compared to 968TB anticipated but the situation is improving. • 0.111: GridPP tape storage made available to LCG/EGEE. At present the tape storage is being used but not really via the Grid route. • 0.112: Fraction of available KSI2K used in quarter: at present a rough estimate shows about 42% of the available CPU was used compared to a target value of 70%. • 0.113: Fraction of available disk used in quarter: This is estimated at 64% compared to the target of 70%. • 0.114: Fraction of available Tape used in quarter: This is estimated at 61% compared to the target of 70%. • 0.131: Tier-1 service disaster recovery plans up to date: This has not been updated within the last 6 months. • 0.143: Accumulated scheduled downtime in the last quarter: The current value of 418 days is almost identical to the current) target of 411 days. The metric expects the 25% figure to reduce to 5% by the third year. • 3.6.3: LCG Deployment evaluation reports: first report due in March 05 was delayed to the second quarter. • 5.2.4. Tier-2 Hardware realisation: This flags the same issue as 0.106 and 0.108 above. Tier-2 hardware has been delayed but the situation is improving. • 5.2.7 Quarterly reports received within 1 month of the end of the quarter: The 05Q1 reports were received late. Some of the delay was due to the unfortunate timing of EGEE meetings. • 6.2.11: Non-HEP applications tested on the GridPP Grid (submitted via the NGS submission mechanism). The NGS submission mechanism is not yet adequate. Oversight Committee

  20. The “Get Fit” Plan • … not (yet) “The Final Solution” • We hope this drives the right behaviour • Plea bargaining is (probably) OK.. Oversight Committee

  21. Some Problem Solving Strategies Oversight Committee

  22. Problem Solving andImproved Communication • “Communication, in essence, is the shift of a particle from one part of space to another part of space. A particle is the thing being communicated. It can be an object, a written message, a spoken word or an idea. In its crudest definition, this is communication. • This simple view of communication leads to the full definition: • Communication is the consideration and action of impelling an impulse or particle from source-point across a distance to receipt-point, with the intention of bringing into being at the receipt-point a duplication and understanding of that which emanated from the source-point..” • from The Scientology Handbook This may be a clue to how we will overcome our problems But we can always improve this.. Oversight Committee

More Related