Experience with the ATLAS LAr optical link

1. Experience with the ATLAS LAr optical link The ATLAS LAr optical link system. Experiences: in the R&D phase. In the integration phase. In the production and QA phase. In early operation phase. The OTx problem and a possible fix as of now. What I would do for future optical links. Jingbo Ye Department of Physics SMU Dallas, Texas Disclaimer: this talk only reflects my own experience and understanding of the optical link, not the official opinion of ATLAS LAr.

2. 1.6 Gbps PECL 50/125 μm 16 TTL 32 LVDS SMUX OTx GLink TX ADC data delay GRIN fiber 40 MHz 80 MHz 40 MHz QPLL FE -- FEB TTCrx The ATLAS LAr Optical Link. 1 • The G-Link based optical link for ATLAS LAr: A total of 1524 channels installed. SMUX: an ASIC based on DMILL, level translating, 2:1 multiplexing. GLink: COTS, operates outside of spec, specially ordered from Agilent. Water cooled on both Feb and ROD. OTx, ORx: VCSEL/PIN based assembly produced by IPAS in Taiwan, with COTS driver and amplifiers. All FE components were qualified for 360 Krad. FPGA GLink RX ORx A collaboration of 5 institutions: CPPM, IPAS, ISN, KTH and SMU The project lasted for 6+ years: 1998 - 2004 BE -- ROD 8 channels FPGA GLink RX ORx Lessons from ATLAS LAr Optical Link, CERN, March 2010

3. The ATLAS LAr Optical Link. 2 • The link TX on the FEB OTx • The OTx is a custom assembly with VCSEL in an ST connector. • There were quite a few issues of this assembly in production in Taiwan and QA at SMU. • We are now having problems with the VCSELs at a few percent level. • Overall this single fiber per FEB design does not answer the reliability requirements from physics (physicist), especially in an environment where no access is possible for maintenance. Glink TX SMUX ~ 400 mm Lessons from ATLAS LAr Optical Link, CERN, March 2010

4. ~ 400 mm The ATLAS LAr Optical Link. 3 • The link RX on the ROD 8 channel optical link integrated on one ROD motherboard. Challenges: Heat dissipation; Noise, cross talks; ORx sensitivity. Link optical power budget OTx power –4 to –7 dBm Required at ORx –17 dBm Lessons from ATLAS LAr Optical Link, CERN, March 2010

5. Lessons That I Learned • In the R&D phase. • Giga bit per second serial data transmission is non-trivial, especially in radiation environment, where parts are not readily available from Digikey.com. • Fiber optics and the electrical/optical interface have a lot of issues that need special expertise. • The above two require experts and special equipment. The price is very high if one only has people “train-on-the-job”; and much higher if one loses those “trained-on-the-job”. • A system level design will certainly be much better than “LEGO building”. But this requires people with expertise of such a system to generate a system specification, in particular: • Clock jitter requirement. • Optical power budget, TX power and RX sensitivity. • Reliability requirement and the information from components. • R&D should end with a production ready design and prototype. Extending R&D into production will cause problems in system reliability. Lessons from ATLAS LAr Optical Link, CERN, March 2010

6. Lessons That I Learned • In the integration phase. • Integration of the link with up- and down-stream electronics needs to be investigated before hand. Debugging may be very difficult if that has not been thought about before the integration. • A clearly defined and communicated system specification will help the integration. • A carefully designed mezzanine board may actually help in integration and operation. Lessons from ATLAS LAr Optical Link, CERN, March 2010

7. Lessons That I Learned • In the production and QA phase. • The production volume is very small (about 2000 units for each part). We spent a lot of time to learn the process, such as “burn-in” of the OTx. • We learned the hard way that optical coupling and connector assembly are not a job for amateurs. We should now stay away from connectors such as ST and SC, especially those connectors are put on manually. • We “invented” our QA process that seemed to be okay, except that • We were doing reliability measurements during and after the production. • In the end, we found out that at least one serious issue with the OTx escaped our attention. • Overall QA and system reliability are big issues we often do not actually know much about. A way out: build in redundancy. Lessons from ATLAS LAr Optical Link, CERN, March 2010

8. Lessons That I Learned • In early operation phase. • Do not open your Champagne too soon. • The problem with the OTx: the VCSELs from TrueLight fail at a few percent level in the past ~1 year of operation. • We tried many things (humidity, magnetic field, ESD, EOS, defects in VCSEL fabrication) to understand the cause of that. This exercise is beyond our knowledge, equipment, and often very expensive. As of now, we still do not have a definite answer to the cause(s) of our problem, although we excluded several things and narrow down to ESD, EOS and fab. defects: Please see Stefan Simion’s talk. • Lesson: HiRel is not easy. Unless you are very confident about all the parts, a design with redundancy may be a way around. Lessons from ATLAS LAr Optical Link, CERN, March 2010

9. Lessons That I Learned • Yet we at least have one variable that seems to have strong correlation with the failure mechanism: the VCSEL optical spectrum (OS). • So far all failed VCSELs (after we started to measure the OS) have narrow OS. J.Parsons S.Simion Lessons from ATLAS LAr Optical Link, CERN, March 2010

10. Lessons That I Learned • We know ESD produces narrower OS. • We have not yet understood it from physics point of view. • We are waiting for the time to tell us what to do: • Replace only OTx with narrow OS. Or • Replace all OTx with a new VCSEL, and different driving circuit. • Replace all OTx with a new design of dual optical channels for redundancy. Reverse ESD at 200V, 300V, 400V and 500V Forward ESD at 500V, 700V, 900V and 1100V Lessons from ATLAS LAr Optical Link, CERN, March 2010

11. The Dual OTx, a Fix With All the Constraints we Have Now. The present OTx: The proposed dual-VCSEL OTx: Lessons from ATLAS LAr Optical Link, CERN, March 2010

12. Dual OTx Test Results .1 Lessons from ATLAS LAr Optical Link, CERN, March 2010

13. Dual OTx Test Results .2 Lessons from ATLAS LAr Optical Link, CERN, March 2010

14. Some Thoughts for Future Links • Multi-Giga bit serial data transmission is not trivial. We should not lose the expertise we now have. The entry price is very high. • We do have special requirements, such as radiation tolerance, no accessibility for maintenance, etc, that cannot be met by a commercial product and design. But this does not mean that we should always go for a custom design and build. We should only do what we can do, and what we are good at, and “borrow” as much as possible from industry. This is the concept/beauty of the Versatile Link project. Lessons from ATLAS LAr Optical Link, CERN, March 2010

15. Acknowledgements Many work is actually done by Andy and Kent at SMU. All credit should go to them. If any statements here are wrong or offensive, I take full responsibility. Lessons from ATLAS LAr Optical Link, CERN, March 2010