RT2010 & xTCA workshop a (personal) summary

1. RT2010 & xTCA workshop a (personal) summary Markus Joos CERN-PH/ESE

2. xTCA • ATCA • A success in the telecom industry (market volume ~109 $) • 80% of the boards use GBE in the fabric network (15% PCIe) • Workshop had a focus on control (IPMI, S/W) aspects • AXIe (http://www.axiestandard.org) • uTCA / AMC • Mid-size (4HP) most popular AMC height • Nothing revolutionary new • General • Little information on real-life experience with HA solutions (technology still too young?) • Performance • No much about actual transfer speeds and latencies of e.g. GBE or PCIe at system level • Recommendation for PCIe: Avoid “reads” (use remote write)

3. uTCA / AMC for physics • RTMs defined • ADF/ZD connector, 20 (compact AMC) or 30 LVDS pairs • RTM ~same size as AMC • RTM gets 12 V (+3.3 V) from AMC • Power: AMC + RTM = 80W (recommended: 50 + 30W) • AMC + RTM = 457 cm2(6U VMEbus = 345 cm2) • Mechanical key • AMC / RTM panels according to rugged uTCA spec (insertion force) • Special backplane (zone 1&2) in preparation • Backplane in “zone 3” possible • Two types of clocks: • High precision radial (TCLK[A/B] as defined in AMC.0) • Low speed bussed (ports 17-20) • Still some issues to be resolved. (e.g. Control of RTM fans) • First products available: • Elma, Schroff: crates • SIS8300: ADC (10 channel, 16 bit, 125 MSPS) with RTM

4. uTCA backplane for physicsTaken from draft document (https://xtca-for-physics.web.cern.ch/xtca-for-physics/MTCA.4.pdf): NOT YET APPROVED • Ports 17 – 20 should be wired as a bus • FCLKA (Clk3) should be used for PCIe clock distribution as defined in MTCA.0 R1.0 (main MCH only) • TCLKA and TCLKB should be wired from the main MCH to all AMC slots • TCLKC and TCLKD should be wired from the redundant MCH to all AMC slots • A possible configuration: • Port 0 should be used for base Ethernet interface • Port 1 is connected to the optional (redundant) MCH • Port 4 to 7 should be used for PCIe • Port 12 to 15 may be used for application specific wire-ring

5. xTCA for physics • ATCA • Zone 3 connector (for RTMs) defined • 3 ADF/ZD connectors (40 LVDS pairs, 5 GHz) for data transfer • 1 combined connector for: • Power: 12V, 50W (+ 3.3V management power) • Management (I2C, JTAG, card detection) • Mechanical keying • Definition of timing signals via Zone 2 connector • For dual-dual star topologies • xTCA S/W • No physics standard planned but recommendations in preparation • Mechanical standards: 90% done • Specification for clock distribution: 90% done • Submission to PICMG for approval planned for September • Questionnaire circulated to identify current xTCA projects

6. xTCA users & products • ATCA • Several plasma (fusion) experiments • ITER not yet fully convinced, will also evaluate PXIe • XFEL LLRF control system • Panda, BELLE II experiments • PCIe switch blade developed by IST (Lisbon) • uTCA / AMC • XFEL LLRF control system (non redundant) • DESY did a physics compliant, general purpose AMC with RTM • Offer to share design or sub-systems (e.g. MMC)

7. xTCA projects mentioned @ RT2010

8. IPMI • Nobody likes it; everybody uses it • PICMG have added features to IPMI for xTCA systems • State machine for hot swapping • Power budgeting • E-Keying • Cooling policy (fan control commands) • F/W upgrade • IPMI spec (600 pages) very ambiguous. This often leads to interoperability problems as manufacturers interpret the spec in different ways • DESY did a “dummy” ATCA AMC carrier as a platform for their IPMC development. They have also built a real-time IPMI analyzer

9. IPMI design Tips Taken from slides of Pete Hawkins (Performance Technologies) • Always Provide a Debug Interface • Serial Port and/or JTAG • Temperature Sensors • Remember Power and Ground Planes Conduct Heat • PCB mounted sensors don’t measure ambient temperature, but that is how everyone implements them • Headroom is More Important Than Absolute Temp • Max Temp – (Ambient + Conduction + Heat Dissipation) • Thresholds • Don’t be Overly Aggressive • “Non-Critical” thresholds are intended to just be informative, but people tend to think they mean something is wrong • Be Especially Careful with “Non-Recoverable” Thresholds • Management software may take action to prevent damage –i.e. may turn off power to a FRU • Especially true for temperature sensors

10. IPMI pitfalls Taken from slides of Pete Hawkins (Performance Technologies) • Management Power Budget • 15W for ATCA Boards, which includes AMCs for carriers and conversion losses • 150mA for AMCs • In-Rush Current • Applies to Management Power and Payload Power • AMC.0 limits must be followed • IPMB Capacitive Loading / Hot Swap Glitches • Follow the PICMG specs carefully • IPMI Controller Resets • Payload Function cannot be affected • Some I/O needs to be latched • Payload Controls, E-Keying Controls, LEDs, etc. • Power Boundaries • IPMI Controller can be powered while payload is not • Connections between payload and IPMI controller may need isolation • PCIeReset / FCLKA • PCIeRequires PERST# to remain active for 100 μs after the reference clock is stable • FCLKA is Part of AMC edge connector, but PERST# is not • Follow Requirements in AMC.1 R2.0 carefully • 0000h vs. FFFFh • IPMI often uses 0000h to mean first item and FFFFh to mean end of list • Programmers are used to using null pointers to end lists

11. Other Information • Exhibitors (only ~5) • CAEN • still not convinced by xTCA • also hesitating to go for VXS • VMEbus 2eSST in the pipeline • Agilent • VXS based 6U high speed ADC (RocketIO on VXS) • Schroff / Elma • “uTCA for physics shelves are useful not only for physics” • Other conference highlights • As many new VMEbus designs as xTCA modules • Use of GPUs in trigger systems • High precision timing systems • High speed ADCs

12. Links & events • xTCA Interoperability Workshop • Week of October 18, 2010, Schroff Training Center at Straubenhardt, Germany • xTCA workshop slides • http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/ • RT2010 presentations • http://portal.ipfn.ist.utl.pt/rt2010/

13. TWEP2010 & xTCA work group meeting a short (personal) summary

14. xTCA for physics • Standards (PICMG 3.8 & MTCA.4) still in draft status but submission for ballot may happen soon • Still a lot of work to be done on the S/W side (such as replacing licensed S/W by open source code) • Apparently there are also efforts to standardize ATCA RTMs in Telecoms • Wiener also planning to develop a uTCA (MCA.4) shelf Other observations • uTCA proposed for XFEL, FAIR and Linac upgrade @ SLAC • Lots of good results from signal integrity tests at xTCA frequencies • Still lots of VMEbus based designs (also for long term projects) • One VXS card (GANDALF for COMPASS) but nobody really compared VXS and uTCA / ATCA

15. xTCA projects mentioned @ TWEP 2010

16. xTCA Work Group meeting • J. P. Cachemiche • MMC soon ready (stable) for use in other projects • Modifications for MMC F/W of DESY and CPPM not synchronized • CPPM willing to manufacture boards • Request for CERN to manage user support • R. Larsen • New requests for PICMG 3.8 (RTM) functionality slow down completion of the standard and may lead to a proliferation of connector types • Still discussions on timing distribution in ATCA • Discussions on a standard way of reconfiguring FPGAs in radiation environments (may require fast LAN add-on to IPMI) • G. Iles • The PCI clock makes the uTCA backplane non symmetrical. Apparently the MTCA.4 team did not spot this • Meeting results • Participants agreed to found a “xTCA Interest Group” • Plan to meet twice per year (during appropriate conference or via EVO) • The xtca-news@cern.ch e-mail list will be used for communication • A Web site (https://twiki.cern.ch/twiki/bin/view/Sandbox/XTCAInterestGroup) has been set up at CERN. All developers of xTCA H/W & S/W will be invited to add a few words about their projects