1 / 53

SCT Operation in 2008 - an eventful year of commissioning and first running -

SCT Operation in 2008 - an eventful year of commissioning and first running -. Early Commissioning and Detector Sign-off…………2 First SCT Run in ATLAS……………………………….4 Commissioning without cooling……………………….7 Barrel sector in SR1……………………………………10 Training………………………………………………….13

nicholascook
Download Presentation

SCT Operation in 2008 - an eventful year of commissioning and first running -

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SCT Operation in 2008- an eventful year of commissioning and first running - Early Commissioning and Detector Sign-off…………2 First SCT Run in ATLAS……………………………….4 Commissioning without cooling……………………….7 Barrel sector in SR1……………………………………10 Training………………………………………………….13 Preparation for Beam…………………………………..14 First Beam……………………………………………….17 Run with Cosmics ……………………………….….…25 Final Calibration …………………………………….…43 Schedule - plan for 09 ……………………………..…50 LBNL ATLAS Meeting - Jan 2009

  2. 1700 ENC Avg chip noise vs chip number Recap of 2007 • Barrel signed-off (May) - 5 days commissioning until cooling heaters failure • Cooling crises: heater repair, extension of pipes to make heaters accessible • Integration in ATLAS central of DAQ ,DCS, Monitoring and run with 4 SCT test modules • ROD and Power Supplies upgrades Noise Noise Occupancy 5.7 e-5 All uniform across barrel Defect Channels Number of defect channel / module (1module = 1536 channels) HV test LBNL ATLAS Meeting - Jan 2009 1. Early Commissioning and detector sign-off

  3. Endcap A: Jan 9-20 2008 Endcap C: Feb 8-11,2008 One very leak loops: 13 modules leak needs repair One “moderately” leaky loop OK at low backpressure Endcap A ENC (Temp = 25deg) Outers: 1707 e Middles: 1653 e Inners: 1149 e Short Middles: 1002 e Endcap C ENC (Temp = 25deg) Outers: 1767 e Middles: 1704 e Inners: 1182 e Short Middles: 1013 e Outers Inners Middles Short Middles Endcap Sign-off February 15,2008 we completed the sign-off of SCT endcaps, officially marking the complete sign-off of whole SCT Noise Defect channels LBNL ATLAS Meeting - Jan 2009 1. Early Commissioning and detector sign-off

  4. SCT first RUN • February 18 to 20, 2008 the SCT (barrel) pre-M6 run in ATLAS • Technical RUN mainly to verify integration and do first timing in the ATLAS readout and control and prepare for the next ATLAS M6 combined run • A chance to try global ATLAS integration with a largefraction of the real detector cooled and powered • Used ROS monitoring to count module coincidences for various timing offsets First SCT pit cosmic on 20 Feb LBNL ATLAS Meeting - Jan 2009 2. First SCT RUN in ATLAS

  5. M6 Run • In M6 (March) first operation of SCT in combined ATLAS run • TRT scintillator trigger + RPC • Operation was limited to Barrel by the number of compressors available in the cooling plant • 3 barrel cooling loops off (3x48 modules out of 2112) because running “warm” at high back pressure and these loops have small leaks • Detector operation and Evaporative Cooling/Heater stable for whole period (24 hour operation) • All RODs and HV/LV Power Supplies were available • One day standalone running to test ROD firmware upgrade • Online monitoring including DQMF • First check of the OFFLINE/Alignment algorithms • Few “noisy” module observed in M6 due to individual noisy strips not masked and occasional HV trips LBNL ATLAS Meeting - Jan 2009 2. First SCT RUN in ATLAS 2. SCT First RUN

  6. Tracks reconstructed in SCT • ID + RPC tracks • SCT residuals (No Alignment applied) • ~ 5000 tracks • SCT residuals (GlobalChi2) ~65m • Compares well with results from SR1 LBNL ATLAS Meeting - Jan 2009 2. First SCT RUN in ATLAS

  7. Cooling Plant Failure May 1st, 2008 • 3 of 6 compressors damaged due to prolonged slippage of magnetic coupler between compressor section and motor • In case of slippage for a long time, Eddy currents will heat inner metallic parts and damage (decompose, break) neighboring plastic parts. • In addition excessive wear on piston rings was found • As a result the C3F8 and plant was contaminated by both the incident and, over a longer period of time, by the wear of piston rings • Subsequent analysis of samples indicated that the part of the plant in UX15 was not exposed to pollution coming from the accident. • ~100-200 kg C3F8 were lost • New C3F8 (1000kg) was received and additional 1000kg have been ordered • The damaged compressors were repaired by Haug • The other 3 were cleaned and checked in-situ • Implementation of the measures that would protect the system • An additional buffer tank was installed to prevent any loss of C3F8 • Compressors equipped with slippage sensors. The PLC will then stop a compressor if it starts to slip • New mechanical filters needed to be installed, to protect the plant from the debris emitted from the piston rings and isolate the distribution racks (and the silicon detectors) • Regular maintenance to replace piston rings (~1000hrs interval) LBNL ATLAS Meeting - Jan 2009 3. Commissioning without Cooling

  8. Damaged parts Magnetic Coupling Motor 1450 RPM Burnt plastics, main source of pollution Crack in the crank cover (puts clean and dirty part of the circuit in contact) LBNL ATLAS Meeting - Jan 2009 3. Commissioning without Cooling

  9. SCT Commissioning without Cooling: Off-Detector • HV communication error (bad current readings) that was causing fake HV trips • Often occurs when card masked or interlocked off. Also occurs randomly, ~ 1/hour • For M6, auto recovery implemented as part of FSM • When channel is masked on to recover from the trip, neighbouring channels may start oscillating • Power cycle all 8 channels of an HV board not possible during a run! • Had to suppress current trip in crate controller software (change thresh) • Firmware upgrade for all boards to solve communication problem (May 19) • Oscillation problem remains and will be solved next year with another major upgrade of the system • Started to investigate “death of vcsels” as observed by pixel • TX vcsel array (12-way) sends optical clock and control signals to modules • Situated in DAQ racks in USA15 • SCT barrel tests showed 5 of 1941 TX with no pin current and too low optical power • possibly related to ESD damage during production at the factory • Since M6 had an SCT integration run in ATLAS readout ~ once/month to test new features and improvements of DAQ, DCS & monitoring chain LBNL ATLAS Meeting - Jan 2009 3. Commissioning without Cooling

  10. SCT Barrel Sector in SR1 • Barrel sector refurbishment started at the beginning of March 2008 • 48 modules, 1 cooling loop, B3 • Cooling • Used the C3F8 cooling plant • DAQ Software and Hardware • Detector backplane resistance tests • DCS Software Interlock and Power supply • Environmental sensors: • 4NTCs for the cooling interlock • 10 NTCs for the mechanical and air temperature • 2 Xeritrons + 2 (HIH + NTC)s for RH and dew point measurements • Sector mechanics and service preparation • Reroute of cables for barrel and EC tests • Re-labeling • Fibres were disentangle • Harness installation • Plug-in • TEST • SR1 space and cleanliness • Test box preparation Started operation July 10 Huge involvement in the various areas of the detector construction and good opportunity for students and new post-docs to learn about SCT Used for TRAINING and DAQ development LBNL ATLAS Meeting - Jan 2009 4. SCT Barrel Sector in SR1

  11. Harness installed SCT Barrel Sector in SR1 Harness test before module mounting with loop back cards Harness testing in the box LBNL ATLAS Meeting - Jan 2009 4. SCT Barrel Sector in SR1

  12. Sector with cooling loop inlet/return side LBNL ATLAS Meeting - Jan 2009 4. SCT Barrel Sector in SR1

  13. Training • SCT training started in May • By June 36 people took training courses in DCS and DAQ in ID SCR, Monitoring in ACR • Training shifts with H8 test box (4 modules) • List of trainees • Started with residents (mostly student and post-docs) • List updated with information from GLs (in July) or directly from people • Training Scheme and method to keep track of training progress • End of June started regular training shifts with Barrel Sector in SR1 https://twiki.cern.ch/twiki/bin/view/Atlas/SctSR1Tutorial • DCS and DAQ • 2 people/shift (4h) • Regular Training Courses in DAQ, DCS, and DQM • 1 hour with instructors • Documentation and tutorial at: https://twiki.cern.ch/twiki/bin/view/Atlas/SctCommissioningDocumentation • Average time to complete training 2 weeks • Qualification overlap shift with experts going through a shift check list (later) • Training info, progress, links to documentation at: https://twiki.cern.ch/twiki/bin/view/Atlas/SctTraining • Started to use the ATLAS Shift Booking Tool OTP (August) LBNL ATLAS Meeting - Jan 2009 5. Training

  14. Commissioning before Beam (I) • Cooling plant repair finished on July 14 • Needed to be able to cool down pixel for beam pipe bake-out • No operation of SCT modules before end of bake-out • Beam pipe bake-out successfully completed 29-31 July • This was a very tense operation - on critical path for LHC to open vacuum valves. • ATLAS went to full shift mode on July 28 • Started of SCT commissioning with cooling on Aug 11 • Once we restarted, running the SCT colder than before, had control problems • Some loops only stable with modules powered up • Some loops also needed modules to be configured and clocking. • Restarting the system was taking long time but got better on synchronizing cooling, SCT DCS and SCT DAQ, after many unwanted shutdowns (power cut, etc.) LBNL ATLAS Meeting - Jan 2009 6. Preparation for Beam

  15. Commissioning before Beam (II) • Frantic effort to get the detector into good shape before first beam • Commissioning cooling loops and basic module control on barrel and endcap • In parallel to Pixel commissioning • Started 24/7 mode (operational shifts and calibration) • Radical DAQ improvements needed completion • Intense calibration: Noise occupancy and equivalent noise charge good for most modules • A few modules still untrimmed (EC-C) giving noisy spots • Calibration ENC noise on barrel is ~1560e- • Noise Occupancy is ~4.4 x 10-5 at 150V • Endcap ENC outer/middle ~1650e- • Noise Occupancy outer/middle ~ 5 x10-5 • Inner Modules much lower due to short strip length • Starting Aug 29/30 had SCT on full power and started to adjust opto readout setting first on barrel then on endcaps LBNL ATLAS Meeting - Jan 2009 6. Preparation for Beam

  16. Voltage setting • Decided that it was safe to run the detector at 20V bias during first beam, with electronics on • Threshold raised to 1.2fC. • Well below 100V limit above which implant damage has been observed • Lowest voltage possible where HV and DAQ stable • Hybrid experts consider LV on to be safer • Pixels stayed off • Then we learned that first beams would be stopped just before ATLAS at the collimator • Worries that many particles going parallel to barrel tracks could deposit even more charge (no simulation of this available) • Reduced barrel HV to 5V and left it out of the DAQ • Endcaps set at 20V - with expected lower (and unmeasured) efficiency LBNL ATLAS Meeting - Jan 2009 6. Preparation for Beam

  17. September 10 • Run 87764 event 40050 LBNL ATLAS Meeting - Jan 2009 7. First Beam

  18. People clapping in the ACR LBNL ATLAS Meeting - Jan 2009 7. First Beam

  19. People (and reporters) in the ACR LBNL ATLAS Meeting - Jan 2009 7. First Beam

  20. People at the SCT desk in the ACR LBNL ATLAS Meeting - Jan 2009 7. First Beam

  21. First Beam “Splash” Event LBNL ATLAS Meeting - Jan 2009 7. First Beam

  22. Space points in EC in one “splash” • SCT roughly timed in to crudely optimize hit occupancy in the central time bin • SCT can time in ‘on the fly’ - invoke SCT busy during timing adjustments • Endcaps running at 20V LBNL ATLAS Meeting - Jan 2009 7. First Beam

  23. SCT People in the ACR LBNL ATLAS Meeting - Jan 2009 7. First Beam

  24. More SCT People in the ACR LBNL ATLAS Meeting - Jan 2009 7. First Beam

  25. RPC: mostly “100” hits trigger arrives slightly late by ~1BC MBTS: mostly “001” or “010” hits trigger arrives slightly early After September 19 • With no prospect of beam, but the full ID available with cooling for the first time, everyone was very positive to start commissioning with cosmics (right away!) • TRT just completed change to Xenon active gas • Pixels detector operational for the first time • Started Combined ATLAS Run until 27 October (almost 6 weeks!) • Alternating calibration periods • Understand of how to time-in the various ATLAS triggers, and how to synchronize the SCT • Propagate configuration and calibration results to the offline • Solved several problems (ROD, TX) and calibrated the detector the best we could • Had several stops due to power cuts, interventions, incidents with cooling, but we always recovered and improved our recovery procedures • Shifts were filled, training in progress and people continued to be active and involved 24/7 • Lots of cosmic events, with and without solenoid on LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  26. Number of SCT tracks • Over 1 million SCT tracks • ~250 thousand tracks with at least 1 pixel hit 200K solenoid off 800K solenoid on LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  27. Cosmic event • Cosmic ray event • Solenoid on LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  28. Before After Problems solved in October (I) • SCT efficiency degraded over the course of a run • Much more obvious as ATLAS stability improved and we recorded many-hour long runs • Observed an increasing number of L1ID and BCID errors • By the end of some runs up to ~22% of modules showed errors • Also saw a decrease in the number of tracks over time (in a run). • Synchronization errors • Once a module got out of synch, its efficiency was zero • John Joseph (arrived on Oct 15) traced the problem to inconsistent treatment of timeouts in the FPGA code ...…..and FIXED IT! • “Data” is taken after BCID problem fixed • “Reference” from a long run, so many modules gave BCID errors by the end LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  29. Problems solved in October (II) • Already started to see signs of TX failures last July. • The number of working TX channels had been gradually deteriorating at the rate of 2 or 3 per day, reaching ~5% dead channels at the end of October. • Original intention was to replace the defective plugins from the batch of50 new plugins received from Taiwan, but the new plugins failed the QA at Cambridge • Made the decision to replace TX plugins with existing (old) spares: • Replaced 8 BOCs, 22 plugins in 17 different BOCs (completed on Nov 12) • Work took 2 days to complete, 25 BOCs needed extraction BOC BPM Chip VCSEL array VCSEL converts electrical signal into light and sends it to the on-detector side. LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  30. SCT efficiencies from later runs • More fixes for conditions and holes on tracks algorithms • Apply a track incidence angle cut • Efficiency for barrel ~99% • More to be understood, especially for the end caps Barrel EC A EC C LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  31. Data Quality Offline Web tools • SCT bytestream check webpageNoise & Error Summaries LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  32. SCT Barrel Residuals • SCT (Barrel) residuals ~20-30 m LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  33. Preparation for ID Combined RUN (5 days - TXG WE) • Detector conditions • Tx replacements and fine opto-tuning • From 200 modules to ~10 • TX channel death rate still 1-2/day • About 70 modules excluded (over 4088 - HV, LV, Ipin=0, Cooling, Misc.) • Including the 36 of ECC • DAQ fixes • one ROD replaced • Calibration and Conditions DB • Online monitoring • more histos • Stability • Shifts • TDAQ Run Control tutorial (~10 people) • RPC (DCS) tutorial • RC+SL shift plan • discussed among run coordinators • In addition to regular shifts • RC and SL shifts in OTP • SCT Detector Operation • DAQ + DCS and DQM • DQM • On call: DAQ, DCS, DQM Also: DSS tests as an exercise to prove that the detector and the cooling can be turned off under different circumstances LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  34. ID Combined Run List Recovery due to mixed-water cooling failure } TRT +RPC } TRT only Recovery due to power cut in US15 { Bad configuration Runs TRT +RPC Good Run } HV=50V TRT only Special runs HV=20V Run transition test LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  35. Tracks and rates Runs with >300 time-out errors after power cut no good configuration LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  36. Time Bins Run 96903 • Mostly “010” followed by “011” hits Summary Time Bins Barrel “010” “011” ECA ECC LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  37. Mean Noise Occupancy for all runs LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  38. Noise Occupancy (Barrel) Run 96903 LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  39. Hit Efficiency Barrel Efficiency depends on layer ECA ECC LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  40. Special Runs at 50 V and 20 V • ~0.5% drop in efficiency for 50 V • ~37% drop in efficiency for 20 V (barrel 4 and disk 2) @ 50 V ECA Barrel ECC @ 20 V (1.2 fC) LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  41. Shifts • Detector was operated by shifters and experts on call (and present) • DAQ, DCS, and DQM tasks (in OTP) over 3 shifts: 7h-15h, 15h-23h, 23h-7h • The last week of running these three shift tasks were merged in one: • SCT Operation task • Plus 2-3 experts on call shifts • Plus 1 shifter for DQM (75% of the time) • DQ checks Offline shift 7h-15h • Overlap shifts • qualification • before first shift period • After Oct 27 (back to commissioning) • DCS watch shifts 24h • Calibration (DAQ expert) day-time shift • Daily meeting at 9h15 • Nov 26 - Dec 1 ID Combined Run • SCT Operation + DQM shifts • Training continued until ~end of November • SR1 was unavailable “only” for 3 weeks for cooling plant maintenance in October All about shifts information at:https://twiki.cern.ch/twiki/bin/view/Atlas/SctShifterInfo • Daily meetings at 15h during cosmic run (including weekends) to discuss current issues between shifts and with data quality offline shifter. At 9h15 after Oct 27. >5 months LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  42. (Include experts) Shifters statistics • 30 residents at CERN plus ~ 1/2 of experts residents SCT trained (overlap) > 2000 shifts SCT Institutes (34): Academia Sinica, Bergen, Birmingham, Cambridge, CERN, Cracow-AGH, Cracow-INP, Freiburg, Geneva, Glasgow, KEK, Kyoto, Lancaster, LBL, Liverpool, Ljubljana, Manchenster, MPI, NIKHEF, Okayama, Osaka, Oslo, Oxford, Prague-CU, Prague-CTU, QMW, RAL, Sheffield, Tsukuba, UC-Santa Cruz, UCL, Uppsala, Valencia, Winsconsin LBNL ATLAS Meeting - Jan 2009 8. Run with Cosmics

  43. Final Calibration in 2008 After ID combined run: • Full SCT ON • Adjusted readout configuration • Latest list of number of modules in default readout configuration • Response curve of SCT barrel and endcaps • Update with tuned thresholds based on gain curve measurement • Expected to improve noise occupancy • Opto survey after end of calibration • SCT sections ON to look at systematic dependencies of SCT Noise as reference for future studies • Noise versus HV scan • Done on barrel side C and Endcap A • Influence of FSI on noise (and leakage current) • Done on barrel at different power levels • Noise versus cooling temperature: bp settings 3.0 (normal), 2.5, 2, 1.5, 1.2 bara • Done on ~500 barrel and ~500 endcap modules • DAQ improvements for calibration: • Speed (faster data archival and faster response from online GUI) • Robustness (found/debugged reason of ROD failing to complete a scan due to errors) • Faulty HV and LV boards replaced SCT OFF December 12 LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  44. Modules in Default Readout • Barrel modules in readout 99.3% 14 of 2112 removed • Endcap A modules in readout 98.7% 13 of 988 removed • Endcap C modules in readout 95.3% 10+36 of 988 removed 98.9% 10 of 952 with loops 1C-TR and 9C-TR removed from statistics Modules excluded LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  45. Noise versus bias voltage • Obtain reference of noise dependence (detector capacitance) on bias voltage • Measured ENC noise with 3ptgain scans at 150, 120, 90, 70, 60, 50, 40, 30, 20 V bias voltage Barrel module with 12cm strips Endcap Inner with ~6cm strip LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  46. Noise occupancy at different voltages • noise occupancy scan (new thresholds) in calibration mode at 150, 50 & 20V 50V 20V 150V LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  47. Noise on barrel with FSI laser running • Tested detector leakage current up to max 200mW (fibre 0) • Tested barrel NO without and with 70mW (using chopper) • Using old thresholds and asynchronous L1A trigger • Tests to be continued with FSI and L1A synchronous No change in noise occupancy 70mW with 50Hz chopper FSI Mean = 3.755 x 10-5 No FSI Mean = 3.748 x 10-5 LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  48. ENC noise in calibration Endcap noise ~ 1650 e- (outer/middle) and inner ~1100e- (950 short middle) Barrel Noise ~ 1550 e- LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  49. Noise Occupancy - New threshold setting • Noise occupancy in calibration mode with new thresholds applied to chip configuration • New noise occupancy is ~2x 10-5 • ~ factor 2 improvement on barrel and less tail! Rms of vt50 point shows modules are well trimmed Mean ~2 1/2 Barrel Previously ~4x 10-5 Old thresholds Full Barrel Endcap A LBNL ATLAS Meeting - Jan 2009 9. Final Calibration

  50. Schedule January/February • Redesign of pipes around the compressors to be sufficiently strong to avoid fatigue cracks induced by vibrations. • All compressors sent back to company for this rework. • Refurbishment of distribution racks in UX15 (until end of April) • Cooling will be off and SCT powered down with exception of DCS/DSS and environmental monitoring. • TDAQ related development work will resume mainly in SR1 • Power supply tests • HV • PP3 debugging • LV pin current measurement (19.01 -- 3 weeks) • FSI tests • Interlock system • Last connections in the DSS system (Laser alarms + ATLAS emergency off signals) • Replacing plug-ins • Work on ID sensors and repair a few damaged pins in the Type II cables • Opto tests • Training in SR1 can resume as early as mid/end of January (3 weeks interrupt in February) and then again from March LBNL ATLAS Meeting - Jan 2009 10. Schedule 2009

More Related