1 / 22

MP3 report : W27 to W33 (22 slides, 10’ ?)

MP3 report : W27 to W33 (22 slides, 10’ ?). Technical issues Organisation update. J.Ph Tock for the MP3 team: TE-MPE : N. Catalan, R. Denz, M. Koratzinos, A. Verweij, M. Zerlauth TE-MSC : B. Auchmann, S. Le Naour, M. Modena, A. Musso, J.Ph . Tock

raymond-sheppard
Download Presentation

MP3 report : W27 to W33 (22 slides, 10’ ?)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MP3 report : W27 to W33(22 slides, 10’ ?) • Technical issues • Organisation update J.Ph Tock for the MP3 team: TE-MPE : N. Catalan, R. Denz, M. Koratzinos, A. Verweij, M. Zerlauth TE-MSC : B. Auchmann, S. Le Naour, M. Modena, A. Musso, J.Ph. Tock + K Dahlerup, E Nowak, A Rijlart, Z Charifouline JPhT – MP3 18th August 2010

  2. Switches from board A to board B • EICs have the right to and are doing the swaps / Also sending loggingMP3 has to be informed of QPS issues (written in MP3 logbook or email or …) • Will run on board B before next technical stop to identify problematic channels ; only communication is affected and not the protection • A new firmware is under test in the lab so this will become transparent Implementation is foreseen during the Christmas break • Circuits concerned (from MP3 logbook) on > 5 weeks:RCBXV2.R2 (10), RCBXH2.R2(2), RCO.A56B1&2 (5), RCBXV&H1.R2 (3) , RQS.A12B2 (5), RQTL11.L5B2 (5), RQTF.A67B2 (2), RQTF.A81B1 (5), RQTF.A56B2 (2), RQTD.A56B2(1) + some more from OP logbook Out of 2400 boards JPhT – MP3 18th August 2010

  3. Discharge of Quench Heater Power Supply • 15/7 : RB.A45 : Quench heater power supply of B17.R4 dischargesReplacement postponed to the day after with monitoring of the 3 other power supplies • 17/7 : RB.A56 : Quench heater power supply of C34R5 discharges • 20/7 : RB.A12 : Quench heater power supply of C34L2 discharges • 20/7 : RB.A56 : Quench heater power supply of B11L6discharges • 23/7 : RB.A45 : Quench heater power supply of C18.R4 discharges • 28/7 : RB.A45 : Quench heater power supply of B8.L5 discharges • 17/8 : RB.A12 : Quench heater power supply of B24.L2 discharges • 7 cases in > 5 weeks – MTBF is 106 hours - > 6000 QHPS in the LHC • Fault identified ; some replacement will take place. The availability should be improved by a factor of at least 10 The procedures were reviewed and agreed. We should not start a fill without redundancy but we may allow to loose redundancy during the fill. So, if an HDS is lost when there is beam in the machine, nothing will happen. However the injection permit is lost and no new beam will be injected … except if QPS_OK masked Was not applied yesterday … QPS_OK was masked (no mention in logbook) for injection so relying on the operator to check if no more QHPS discharges. Software will check this automatically and initiate (after EIC validation?) a slow power abort if 2 dipoles or 1 quadrupole QHPS is discharging. Implementation planned for October TS JPhT – MP3 18th August 2010

  4. Global BB detector : mains • Spurious trigger signal not related to any real event which is self curing (FILLING BUFFER --> FILLING BUFFER NOT OK --> FLT TIMEOUT --> LOGGING --> QPS-OK back)The origin of the temporary problem is related to the Global Busbar Detector, not filling the buffer, is related to a time-out and will last for 2 to 5 minutes. It has an automatic reset (6/8, 9/8,..) • In case of loss of QPS_OK, waiting 5 to 10 minutes can solve the problem • 15/8: Checking QPS signal in a stable beams phase, it was noticed that QPS_OK was lost for a communication problem. Protection is still ensured. Intervention for a reset of the local main BB detector in the tunnel was planned that solved the issue • Personnal computer to be rebooted locally (access) ; will be solved during Chritmas break • Check allowed to plan the intervention and minimize its impact • There is a plan to remove it during the Christmas break as nQPS is taking care of this. The information on the total voltage across the BB being a useful information, this will be kept but only in monitoring and not in the interlock JPhT – MP3 18th August 2010

  5. Trips due to too fast request • Request functions have been smoothened • 8 RQTD.B2 tripped ; feedback functions (13/7) • Various RQTF/D; real time trims (15/7 & 5/8) • Various RQS; new ramping-up parameters (23/7) • Various RQTF/D due to tune feedback re-enabled by mistake (1/8) • RCBV16.R4B2 (3/8) • RSF2.A23B2 and RSF.A67B2; squeeze functions(12/8) • RCBXH1.R2: Trips due to mismatch (30 mA) of the current before the start of squeeze • Partial watch dog is implemented (OK for preloaded functions) • Limits increased: • RQX.L/R2 & 8 : During setting for the 10A/s ramp, problem encountered with these 4 inner triplet circuits (slightly exceeded the initial commissioning value of 6A/s - needing something like 6.05 A/s). There is no limitation for a slight increase from the converter point of view so the limit was increased to 6.2A/s. Values have been changed in LSA and synchronised into the FGC. Approved by MP3 JPhT – MP3 18th August 2010

  6. Undulators • Status regularly checked to plan for a reset, minimizing operation disturbance • No perturbation in the last 6 weeks • Autonomy > 10 days Triplets circuits • Many trips due to PC when ramping down • Triggers MP3 verifications of the heaters firing • EPC issue under investigation to assess if solution is possible JPhT – MP3 18th August 2010

  7. Communication problems with access • B18R5 old QPS; Fip com lost; access required; incorreclty assembled connector (8/7) • Segments 8R4 & 15 R4 of the mains: WorldFip experts accessed tunnel (10/7) • RB.A12 : No communication with current leads; Access to tunnel to restart main BB detector (12/7) ; will not be used any more next year • S56 global BB detector (26/7): reboot from tunnel ; will not be used any more next year • DQAMGS.B11L5 : needed MPE piquet access in tunnel (10/8) JPhT – MP3 18th August 2010

  8. Communication problems without access • Communication lost on DQAMC.21L7 S67: cured by resetting the gateway(8/7) • RCBV10.R1B1 : PC communication (8/7) • RQT13.L6B1, RQT13.L6B2, RCO.A56B1 and RCO.A56B2 : (Fast) Power cycled cured it (8/7 & 15/7 & 17/7) • RQTF/D in 78 : Power cycle cured it (10/7) • RCO/D.A56B1 : communication lost on crate; Power cycle cured it (12/7) • RQRF.A67B2 LD1 com pb (13/7); Power cycle cured it • Network in CCC; lost QPS_OK of 6kA&600A in34 not visible; masked as at injection level ; came back < 15 minutes later (15/7) • S67 communication lost with gateway; cured by rebooting gateway (15/7) • RQD/F.A12 : Lost communication with MQ.16L2 (DQAMC.16L2) while ramping up the current. By miracle, communication came back after 25 minutes (23/7) • RQT13.L6B1&2, RCO.A56B1&2: Loss of com. with the crate ; cured by power cycle (4/8) • S67 mains lost due to communication problem(8/8) • Loss of QPS signals in S12 ; server had to be rebooted (12/8) • Loss of Loss of communication with main BB detector of RQF/D.A67 cured by sending logging (EIC) (15/8) JPhT – MP3 18th August 2010

  9. External? electrical perturbation (1/2) • 10/7 : S67&78 lost: origin RQ10(9).R6 & L8 • No PM of RQ9, as on same acquisition system as RQ10, will be solved after Christmas break • Long cables and specific settings (see further) • 13/7 : RQ5.L2 • Trip du to noise on U_RES ; QH discharge is OK ; During a ramp ;Noisy board that was replaced JPhT – MP3 18th August 2010

  10. External? electrical perturbation (2/2) • 23/7 : RQ9.R2 • Trip ; heaters fired ; U_Res over threshold but origin of perturbation not identified • 27/7 : RQ6.R1 • Trip ; heaters fired ; U_Res over threshold but origin of perturbation not identified ; • typical analysis with heater discharge and recharge checked JPhT – MP3 18th August 2010 JPhT – MP3 18th August 2010

  11. MP3 is 1st line for analysis Some examples: • 16/7 : RQT12.R5B2 : Trip due to fan problem on PC • 18/7 : RQ6.R5 : PC trip (EPC issue) • 25/7 : RCS.A45B1 : Tripped 3 times; EPC changed a power module • 1/8 : RB.A56 : beam dumped by RB trip due to water fault issue in the PC. EPC intervened • 7/8 : RQ8.R6 : Trip first thought to be due to noisy QPS board but finally was due to a PC component that has been changed • 8/8 : RCO/D.A56B1 : Trip (<1A) during the night ; indication that possible earth current fault ; EPC intervened • 16/8 : RB.A56 : Cannot be rearmed due to water fault so intervention by CV • 17/8 : RCBH30.L2B2 : Circuit tripped due to a VIN DC UNDER VOLTAGE voltage • 17/8 : RQT13.L2B2 : Tripped while precycling due to the PC with an internal voltage reference fault JPhT – MP3 18th August 2010

  12. 8th of July • RSS.A34B1Self opening of the switch (#3) ; first event of this kind in the LHCThe faulty switch Z of the EE system has been replaced on 19/07Circuit not used in between • RB.A81Trip du to nQPS board B21R8 (was changed- Broken resistor on the board) JPhT – MP3 18th August 2010

  13. 22nd of July • RCBCV7.L2B2 and RCBCH7.L2B1 • Test following intervention on the PC side • Resistance measured during PNO.A1 is slightly larger than the tolerance. Already observed in the hardware commissioning but very stable. • To be followed by MP3 JPhT – MP3 18th August 2010

  14. Not really MP3 but safety issue : 23rd of July : RB.A78 • Trip of RB.A78 • Sector trip when switched ON at 08h18m18s. To get access in L8, the EiC has put the RB on OFF without first requesting STANDBY. The result was that the current in the RB was SLOWLY decaying towards 0 (see timber plot). Access was then given with current in the RB (about 500 A) although the EiC thought there was 0 A. After access was finished, the EiC put the RB on standby. This created a huge voltage in the circuit because there was still a current of about 200 A in the circuit at the standby command. Then the main busbar detector tripped because U_RES went above the threshold and the switches were opened. • Sequence was not followed • This is managed in another body Sector trip with 200 A Access with 500 A in the RB JPhT – MP3 18th August 2010

  15. Monday 3th August • Beam dump triggered by QPS • The trip was caused by the spontaneous openig of a circuit breaker powering the QPS detector in 18L3. Intervention was required • 2nd occurrence this year JPhT – MP3 18th August 2010

  16. Tuesday 27 July • RCBXH1.R2 U_res signal with a significant offset tripped during pre-cycle • Required an access at 02:00 to be reset from the tunnel ; will be investigated at next TS JPhT – MP3 18th August 2010

  17. Q10-Q9 sensitivity (to perturbations) • Special configuration applied to these magnets to measure higher range voltage • Poor resolution on the low voltage range • High sensitivity to noise of the triggering signal • Restoring the nominal settings will imply removing two resistors for each card. • MP3 recommended to test in the lab and implement 1-2 examples during the next technical stop. • Activity on-going JPhT – MP3 18th August 2010

  18. IPQs bus-bar resistance measurements • Measurement of bus-bar resistance for the 6kA circuits • Reference values for time evolution and nQPS upgrade • Already 14 magnets measured in the dispersion suppressor • All values around 5 nOhms for 4 to 5 splices. • Measurements require special equipment displaced and connected during the technical stops. JPhT – MP3 18th August 2010

  19. Parallel activities • Hardware procedures and test plan for next shutdown/break under revision • New tools on preparation • New interlocks that may lock the circuits after automatic analysis of the data • 1st version to be released for September 2010 • BLM threshold adjustment tests • A dedicated MP3 meeting will be organized for this subject. Location was already identified but needs to be confirmed. JPhT – MP3 18th August 2010

  20. MPE interventions at next Tech Stop • IPQs: New system for Q10R2 (Maybe Q9 & 8 too) • Will require a full re-commissioning (Thursday night?) • Change of switch on QHPS : • Beginning of a campaign ; will improve availability • Recharging will be monitored / no discharge test planned • Then campaign to be completed during Christmas break • Snubber capacitors (S12 or 78) • Will be installed but not connected • Connection during the Christmas break • Intervention on critical safety devices • Dedicated tests have to be foreseen, with powering JPhT – MP3 18th August 2010

  21. New supervision mode from MP3 (Since 28th of July 2010) • Two persons on call for LHC operation • 7:00 to 19:00 on weekdays • 19:00 to 7:00 on weekdays and 24/24 on weekends • Attendance to the 8:30 and 9:00 meetings • Close follow-up of the LHC operation • Shifts will be organized after technical stops • Same commitment and workload for the MP3 members but resource optimization x162266 JPhT – MP3 18th August 2010

  22. MP3 Actions Average: 2.1/day, stable (2.0 over the last 30 days) Technical Stops –Recovery from TS Switch to supervision mode JPhT – MP3 18th August 2010

More Related