1 / 4

Trigger Frontier tests at BNL and MC Production Issues ADC weekly CERN June 26 , 2012

Trigger Frontier tests at BNL and MC Production Issues ADC weekly CERN June 26 , 2012. Alexei Klimentov Brookhaven National Lab. Trigger tests. Trigger tests were conducted o n Jun 14 th to evaluate trigger reprocessing at BNL

beate
Download Presentation

Trigger Frontier tests at BNL and MC Production Issues ADC weekly CERN June 26 , 2012

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Trigger Frontier tests at BNL and MC Production IssuesADC weeklyCERNJune 26, 2012 Alexei Klimentov Brookhaven National Lab

  2. Trigger tests • Trigger tests were conducted on Jun 14th to evaluate trigger reprocessing at BNL • Tests caused failures in MC production for critical Higgs sample needed for ICHEP • Use of Frontier was enabled by setting site wide configuration parameter in CVMFS configuration • Production on 70% of ATLAS Grid sites was affected for ~48h (Jun 14 11:05am – Jun 16 12:40pm) • The problem was made public to the collaboration on Jun 20th early afternoon after Higgs group reported that there is no trigger efficiency information in the files produced by the ATLAS Production System • The decision about tests was made by a small group of people w/o discussion with the Operations and ADC Coordination • The decision to change a parameter was made after e-mail exchange Alexei Klimentov – ADC weekly

  3. Timeline Jun 14 11:07 : Variable TRIGGER_USE_FRONTIER was set for the Trigger reprocessing on the Grid Jun 15 • 11:21 ADCoS submitted the first ticket sr #129523 • 16:30 ADCoS submitted the second ticket sr#129540 • 18:54 FRONTIER support was notified • FRONTIER team supposed that jobs are badly defined • 21:35 Trigger expert was asked to have a look at the FRONTIER failures June 16 • 04:33  First ticket sr #129523 wasreassigned to FRONTIER bug #95442 • 05:52 Trigger expert wasasked if this bug couldberelated to the TRIGGER_USE_FRONTIER variable • 12:37 Variable TRIGGER_USE_FRONTIER wasremovedfrom the sites June 18 • 10:14 GDP expert debuggedthat FRONTIER failureswereindeedrelated to the TRIGGER_USE_FRONTIER variable June 19 Physicistsreportedwrongresults in the ICHEP samples June 20 • 11:42 Trigger expert confirmed the bug in the Trigger DB access code • ADC Coordination wasinformed about the incident • 14:00 – 16:00 brainstorming meeting to understand how severeis the problem and how to minimize an impact on Higss production • DDM operationschecked the list of affected files (all sites hosting LFC repliedpromptly) • Rod’sproposal to re-generate the wholesamplewithoutdeletingaffected files from the datasets • 21:19 The list of affectedsamplesprovided to physicists June 21 • 23:29 Production managers resubmitted the affectedsamples June 24 Resubmittedsamples are more than 70% done SW&C plenary week Alexei Klimentov – ADC weekly

  4. Summary • The fundamental ADC rule was violated : • NO DECISIONS AND CHANGES WILL BE MADE BASED ON E-MAIL EXCHANGE • Only a small group of experts were involved in the trigger tests planning • ADC Coordination warning was ignored • NO CHANGES WILL BE MADE IN THE RUNNING SYSTEM BEFORE DATA SAMPLES FOR ICHEP WILL BE READY • ADC has no comments concerning software itself, it will be followed by the SW team • Many thanks to Operations, sites and shifters, due to their hard work during last weekend we were able to minimize the impact of the “tests” Alexei Klimentov – ADC weekly

More Related