1 / 19

Streamlining CDF Shift Operations

Streamlining CDF Shift Operations. Peter Wilson Mary Convery CDF Detector Operations All Experimenters Meeting February 26, 2007. Running CDF thru FY2009. Challenge: collect high quality data with high efficiency as the pool of physicists shrinks! LHC will turn on in FY2008

rigg
Download Presentation

Streamlining CDF Shift Operations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Streamlining CDF Shift Operations Peter Wilson Mary Convery CDF Detector Operations All Experimenters Meeting February 26, 2007

  2. Running CDF thru FY2009 Challenge: collect high quality data with high efficiency as the pool of physicists shrinks! • LHC will turn on in FY2008 • Many people working on CDF also on ATLAS or CMS • Expect fewer Postdocs and students in 2008/2009 • However, we don’t need as many people as in 2001: • Experiment is mature • Systems are well understood • Procedures are well defined • Upgrades/changes are minimal • Effort initiated by previous Ops heads (Willis Sakumoto and Pat Lukens) to re-organize shift operations to use fewer people while maintaining or increasing operational efficiency

  3. CDF Shift Crew - Ops Man Operations Manager (Ops Man) - On call 24/7 • Makes daily operational plan • Typically experienced postdoc - 1 wk on/2 wks off for 6 months CDF Control Room

  4. CDF Shift Crew - SciCo Scientific Coordinator (SciCo) - 8/7 x 3shifts • Leader of the shift crew • Responsible for implementing daily plan • Communicate with MCR • Summon expert help • Provide food for the crew! • Faculty, lab scientific, staff or senior postdoc CDF Control Room

  5. CDF Shift Crew - DAQ Ace DAQ ACE - 8/7 x 3 shifts • “Runs” the experiment • Controls the data acquisition system • Setup run configurations • Start/stop DAQ runs • Monitor DAQ status, diagnose and fix DAQ problems  • Run calibrations CDF Control Room

  6. CDF Shift Crew - Mon Ace Monitoring ACE - 8/7 x 3 shifts • Detector HV monitoring and control. Diagnose/fix HV problems    • Monitor status of low voltage power supplies • Responsible for safety of Si in the event of a cooling alarm • Monitor beam conditions for detector safety • Twelve Students and postdocs cover DAQ & Monitoring Shifts in 10 week rotation CDF Control Room

  7. CDF Shift Crew - CO Consumer Operator - 8/7 x 3 shifts • Data quality monitoring for all detector subsystems    • Ten “Consumers” each receiving a few Hz (of ~100Hz) of events from L3 • Detector hit occupancies and average pulse-heights in minimum bias data • Trigger rates and trigger algorithm validation (compare emulation to data) • DAQ performance and errors • Compare results to references • Check all routine detector calibrations • Any CDF physicist CDF Control Room

  8. CDF Shift Crew - Cryo Process System (Cryo) Operator - 12/7 x 2 shifts • Solenoid and Cryogenic System operation • Gas systems • Cooling systems for electronics and detectors • Electronics rack protection • Four PPD technicians on rotation, one other tech fills in for sick leave/vacations CDF Cryo Control Room (1st Floor)

  9. Streamlining the Shift Crew • Consumer Operator Shifts taken remotely • Pilot operation at INFN-Pisa • Reduce need for travel for overseas members of CDF • “OWL” shift is 7:00-15:00 in Pisa, Italy or 15:00-23:00 in Tsukuba, Japan • Reduce the number of Aces from 2 to 1 • Consolidate responsibilities with the remaining ACE • Transfer some monitoring tasks to Cryo Operator • SciCo trained on some tasks of monitoring Silicon detector • Safety of detector if Ace is not in control room (eg high beam losses) • Automate tasks to allow shift crew to focus on decision making

  10. Remote CO • Effort started early in 2006: • All monitoring plots available via Web • Tools for remote starting and stopping of Consumer programs • CO shift room in Pisa: monitoring workstations and video conferencing • CDF  Pisa Video conference for SciCo-CO communication • Remote CO does all of the same functions as local CO • Experience shared with and gained from US CMS remote operations • Team from Fermilab, Pisa, Tsukuba, UCLA and Waseda led by Kaori Maeshima and Fabrizio Scuri • First official remote shift: week of November 10 • Commissioning in October 2006: simultaneous Remote and local CO • Through June 2007: schedule 1week/month of remote CO from Pisa • Easy to extend to other locations: planning to start shifts in Japan starting July 2007 • Goal: all OWL shifts covered remotely in 2008-2009

  11. Single ACE Operation • Planning started in Summer 2006 • Implementation starting mid-October • Re-arranged control room computer consoles • Added ACNET and HV control next to DAQ consoles • DAQ and Monitor functions from one work area • Automated recovery of Silicon HV crate communication failures, Ace previously left the control room (Hockerization Auto-Hockerization) • Automated generation of Hourly ACNET plots of beam losses • Updated monitoring pages and documentation for Silicon Cooling and LV PS monitoring  Cryo Operator • Add audible alarm for Silicon detector PS current monitoring  SciCo • Improved SciCO training • Team from Silicon detector group, CDF Ops, CDF DAQ and Accelerator (Ron Moore and Brian Hedricks) led by operations heads • First shift with single Ace Dec 29 • Followed 10 days where 2nd Ace acted only as backup

  12. CDF Owl Shift - Feb 15 Ace (Michal Kreps) CO in Pisa (Simone Donati) SciCo - John Yoh

  13. Owl Shift in Pisa - Feb 15

  14. Monthly average good runs w/ Si Efficiency High with Reduced Crew

  15. Experience with Reduced Shift Crew • No significant problems with shift crew changes, biggest challenges have been substitutions for shifters who are ill! • New Years weekend needed to find substitute Ace • Remote CO needed to leave after 6 hours of shift • Operational efficiency has not dropped • With single Ace, crew is operating even more as team • Experience from three weeks of remote CO: as effective as local CO • Automation is reducing downtime: • Recovery of Si HV Hockerization took 10-15min • Auto-Hockerization now takes about 4 mintues • Typically about once per store

  16. Additional Improvements Coming • In case of dangerous beam conditions, detector HV must be set to Stand-By: • CDF TEVMON tool monitors losses, abort gap beam, TevRF, etc based on ACNET variables, warns shift crew of dangerous conditions • If shift crew is occupied might be delayed in responding • New TEVMON based on an Open Access Client (OAC) by Ron Moore • ACNET variables indicate danger level and provide permit for operation of detectors • Automatically set Silicon HV to standby in dangerous conditions • Final testing underway • HV problems sometimes require Ace to leave control room: • Plug calorimeter HV regularly requires procedure to re-establish communications  move control PC and interface hardware from 3rd floor to control room (2nd flr) • Other HV systems need to have programs restarted or PCs to be rebooted on 1st floor  working on remote access to HV PCs from control room

  17. Some Key Members of the Team

  18. Members of the Team Thank you to the many people who helped make this happen: • Fermilab: Bill Badgett, Kurt Biery, Ray Culbertson, Steve Hahn, Brian Hendricks, Jonathan Lewis, Kaori Maeshima, Ron Moore, JJ Schmidt, Peter Wilson • CDF Technical support team: Dervin Allen, Del Allspach, Jim Loskot, Lew Morris, Bill Noe, George Wyatt • Duke: Dean Hidas • INFN Pisa: Dario Fabiani, Luciano Ristori, Fabrizio Scuri, Jose Enrique-Garcia • Tsukuba: Taichi Kubo, Koji Nakamura • Waseda: Junji Naganoma, Tetsuo Arisawa • Rochester: Jennifer Gimmel, Willis Sakumoto • Rockefeller: Mary Convery • UCLA: Charles Plager • Yale: Ulrich Husemann

  19. Summary • CDF has successfully reduced the shift crew from 4 physicists to 3. • 25% of Owl shifts have only 2 at B0 • Process of reducing shift crew has improved the tools for shift takers • Operational efficiency has been maintained or improved during the transition • Additional improvements are coming which should result in more efficient shift operation • We are working on other improvements to increase our operational efficiency and prepare for a smaller physicist support • For example, adapting Electronic Worklists from AD for scheduling accesses and tests

More Related