RHIC PS System New Systems, Upgrades 8/14/14

1. RHIC PS System New Systems, Upgrades8/14/14

2. Upgrades/New Systems • Suncraft PS Upgrade • New E-lens cold magnet ps’s for Run 14 • New bipolar b-qtrim-ps and y-qtrim-ps for Run 15 • New Main ps regulator • VODH system • Quench Detector

3. Suncraft PS Upgrade Plan • Improve reliability of existing suncraft ps’s • Almost every run we have been improving these ps’s. • Kepco Bipolar ps’s • Installed first time in Run 12 • Using a COTS unipolar + H-bridge • Installed first time in Run 14

4. I. Suncraft Improvements • OFF trips eliminated (Implemented run 12) • Adding wires around bad connections stopped voltage drop outs (Implemented run 12) • Many, many other upgrades not listed here implemented over the years • Upgrades still to be done: • Soft start circuit • Double FET resistors • Replace FET fuses • One PC board replacing 3 PC boards

5. IIa. Kepco ps’s-History • Installed two 20V/50A kepcos in yo12-tq6 (150A bipolar) 4/11/12. No failure in run 12 • Installed three 10V/100A kepcos in y2-q89 (300A bipolar) 4/17/13. No failure in run 13. • Run 14-Install kepcos in q89 slots, 1 tq slot • Y2-q89 false kepco fault (5/17/14), BNL connections to 3u control chassis, ps and 3u chassis replaced. • Y6-q89 crowbar faults (4/25/14), BNL connections to 3u chassis and cold solder joint on 1 kepco ps • Y8-q89 installed/removed 5/28/14, problem with paralleling cables discovered. Y8-q89 re-installed 6/16/14. No failure. • Yo12-tq5 installed single kepco 6/16/14. No failure.

6. IIb. Kepco Summary • BNL problems • 3u control chassis and external interconnect harness workmanship issues which led to bad connections causing failures • Kepco problems • One front panel returned for intermittent turn on problem. Discovered in 7W during test. Replaced by kepco. • One ps had a cold solder joint which was discovered after y6-q89 was removed. • Paralleling cables workmanship problems led to ground fault.

7. IIc. Kepco Plan going forward • Complete Inspection of all 3u control chassis wiring and external harnesses. Rework as needed. Re-test. • Hi-pot completed system not just individual ps’s. • New Paralleling cables from Kepco. Properly constructed. • 2014 Shutdown: Install kepco ps’s in 3 service buildings • 2B, 6B and 8B, q89 ps’s only • 2014 Shutdown: if there is time install the rest in • 4B, 10A and 12A, q89 ps’s only • Still want to be cautious.

8. COTS ps + H-bridge • Did not get first H-bridge until January 2014. Ordered Jan 2012 • Installed in bo7-tq4-ps 5/14/14. No failure. • Will be installing COTS ps + h-bridge in b-qtrim-ps and y-qtrim-ps for Run 15

9. Suncraft Upgrade Summary • Suncraft ps reliability improvement seen. • Problems with running kepcos identified and being fixed. Will install in q89’s in 3 buildings (possibly 6 bldgs). Still want to be cautious. Alternative ps source found. • APS h-bridge successful in tq slot. Will be used in b-qtrim-ps and y-qtrim-ps. Still want to be cautious. Nested slot now. Alternative ps source found.

10. E-lens Cold Magnet PS system • Two 20V, 700A unipolar Sorenson ps’s • Four 15V, 534A unipolar Sorenson ps’s • 24 suncraft style correctors, 20V, 50A • 8 QPA’s • 1 Quench Detector • 6 RHIC 3u control chassis and cards • 3 AC compartments • Other control chassis, qpaic’s, node cards, temperature controller, one wire system, ice ball system…

11. E-lens PS system Problems • Possible loose connection problem within blue fringe AC compartment. Causing ps drop out. • Loose connections on new qpa IGBT caused false fuse faults. • Controlling Ice on leads • Fix permit signal to cryo from qd so we can run in auto flow mode • Loose d connector on qd

12. New Bipolar q-trim ps’s • B-qtrim and y-qtrim are unipolar SCR type ps’s. 40V, 300A. • Each will be replaced by two 22V 180 unipolar COTS ps’s connected to 2 linear h-bridges which are in parallel. • 10/1/14 should be ready to test.

13. New Main PS regulator • Carl tested and fixed the new PLL/firing card. • Will be installed in all 4 main ps’s for run 15. • New regulator work continues

14. New VODH system • First used in Run 14 • No bottles required • One problem when a communication fiber was accidentally knocked loose by a technician but it did not trip the cryo valves because a comm failure is not allowed ot trip valves.

15. Quench Detector • Need to start on Quench Detector upgrade

16. Run 14 Performance(if there is time otherwise end here)

18. MTBF Low Energy vs High Energy

20. Leading Problems • Beam Induced Quenches 53.33 hrs • BMD regulator crimp connection 14.98hrs • Kepcos 14.9 hrs • Suncraft 50A correctors, mostly leaky cap 10.68hrs • Suncraft 150A 7.95hrs • QD’s, mostly radiation 7.8hrs, also running very hot in some areas, dead zones or broken A/C-lead to long term problems • QPA’s (IGBT failures mostly) 7.57hrs

21. Down Time in Hours Runs 14-9

22. Run 14 vs Run 13 Start Up

23. 2013 Major Shutdown Work • Kepco installs • Dynapower Gate Drive/Iso amp boards • New bipolar q-trims • Suncraft leaky capacitors and fans • Suncraft 150A/300A upgrades • Investigate/Fix QPA IGBT problem • Main ps New Firing board/PLL board install • Main ps OCC re-work-last one

24. 7W • Still very crowded • Need more room

30. ENDCharts, Stats follow if needed

31. Comparison of other labs MTBF and RHIC

32. MTBF of just ps’s HERA 1996 RHIC

38. Overall MTBF

45. Run 14

47. Backup Slides

48. Why replace the Suncraft ps’s ?(part 1) • RHIC ps’s were the number 1 source of downtime in RHIC run 11. • Suncraft 150A and 300A p.s.’s were the number 1 source of downtime when compared to the other RHIC p.s.’s (or anything else in the group). • We still are sending boards back to Suncraft for repair and still buying spares from them. They still change the micro program for us. We depend Suncraft too much. We have to develop an alternative source for these supplies. • Supplies have been rebuilt many times. Connections, PCB, and components have deteriorated and have proven to cause reliability problems. • The packaging of the supplies is poor and are very labor intensive to work on. • The supplies have operated at elevated temperatures for many years and have started to show end of life failures. (Fuses, Chips, more) • The interconnecting bus design is poor. In addition it has corroded and oxidized due to initial operating environment. • Due to the construction every time the supplies are worked on it can be expected a high percentage will develop new problems.

49. Why replace the Suncraft ps’s? (part 2) • Even with some improved reliability new systematic problems have been discovered in run 11. We will have to commit at least two engineers and two full time technicians to repairing this supplies as long as they are operating. This does not include upgrades or summer shutdowns. • 43% of the 300 ampere power supplies have failed this year. • No commercially available alternative to these supplies is available. We have to develop an alternative source for these supplies. If we do not develop a replacement for these power supplies it is likely that the biggest source of downtime for eRHIC will be failures of the Suncraft 150A and 300A power supplies. • Increase number of spares. Remove repeat offenders from operations. • ~$10M in downtime due to Suncraft ps replacement • Many engineers and technicians have worked on these ps’s (fulltime and part time). Drain on resources. • Do not get complacent. Will this improvement continue with the 150’s? Investment in future of RHIC. 5 Year warranty on COTS ps