1 / 23

MicroTCA in CMS

MicroTCA in CMS. Not Official! Just my opinions. Greg Iles 6 July 2010. My background: Calorimeter Trigger. How can we improve the trigger?. Hadron Calorimeter. Electromagnetic Calorimeter. RCT. 3 Tb/s. GCT. 0.3 Tb/s. Trigger. Requirements for a trigger. Must process Tb/s

luz
Download Presentation

MicroTCA in CMS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MicroTCA in CMS Not Official! Just my opinions... Greg Iles 6 July 2010

  2. My background: Calorimeter Trigger How can we improve the trigger? Hadron Calorimeter Electromagnetic Calorimeter RCT 3 Tb/s GCT 0.3 Tb/s Trigger Greg Iles, Imperial College

  3. Requirements for a trigger... • Must process Tb/s • Not a problem, just make it parallel, but.... • Need to build physics objects, which don’t observe detector granularity! • Data sharing • Data duplication • Need to sort physics objects • Avoid multi stage sort to minimise latency • Restricts number of “physics builders” due to fan in constraints • Only have approx 1us • Each serialisation is 100ns - 200ns Greg Iles, Imperial College

  4. High Speed Serial Link Technology • Pros • Significantly higher data rate than standard I/O • Easily connected to optics • Serial backplanes available: • e.g. MicroTCA: Based on Advanced Mezzanine Card (AMC) developed for ATCA • Also ATCA, CompactPCISerial, VPX • Serial cross-points available • Wire speed duplication of data • 144x144 at 10Gb/s Matrix card: Part of the GCT project Design by Matt Stettler (LANL) Greg Iles, Imperial College

  5. ATCA, December 2002 MTCA.0 R1.0, July 2006 Built around the mezzanine card (AMC) designed for ATCA Greg Iles, Imperial College

  6. ATCA Card from PICMG Short Form Spec Greg Iles, Imperial College

  7. AMC Card Originally intended as hot-swappable mezzanine standard for ATCA but soon used as the basis for the MicroTCA standard • 20 bidirectional diff pairs to at 12.5Gb/s (not yet demo’d) • Normally operates at 3.125 Gb/s • 5 clocks • Protocol agnostic • PCIe, SRIO, GbE • 6 form factors • 74 or 149 mm wide • 13, 18 or 28 mm high • 180 mm deep • Power supply: 80W (max) on +12V • Connector: 85 pin (single sided) or 170 pin (double sided) edge connector Greg Iles, Imperial College

  8. Matt Stettler, LANL pioneered uTCA in CMS μTCA • Best thing about μTCA: Very flexible • Also possibly the worst thing.... • Built around a MicroTCA Carrier Hub (MCH) • System management via IPMI (Integrated Peripheral Management Interface). Uses I2C • GbE • SATA/SAS • Clock distribution to/from slots • Fat Pipe (x4 lanes) • Redundant system possible • Up to 12+1 AMC cards Greg Iles, Imperial College

  9. - 2U / 19” chassis - Slots for up to 12 AMCs - Cooling for 40W per slot - 6 mid size (single or double width) AMCs - AC or DC PSU - Single star backplane MCH - Fat-pipe mezzanines for: - PCIe, 10GB-Eth, Serial RapidIO - Clocks Greg Iles, Imperial College

  10. Dual Star, Telecom Clocks MCH2: LHC-CLK, TTC & TTS and DAQ Concentrator MCH2 12 Full width AMC slots MCH1 MCH1 providing GbE and standard functionality Vadatech VT891 Greg Iles, Imperial College

  11. Reserved Alt DAQ? Alt Comm? GbE SATA or SAS Fast Control (Not SerDes) TTC-Out TTS-In FatPipe x4 lanes e.g. PCIe SRIO DAQ-In or Switch LHC 40MHz Clk Clk Out Clk In Not shown:8 spare ports2 spare clks Dual star Greg Iles, Imperial College

  12. Greg Iles, Imperial College

  13. Towards a CMS system • DTC by Eric Hazen, • Boston University • Purpose • Distribute Clock • Distribute Fast Control • Receive Fast Feedback • Optionally DAQ concentrator • Trigger cards send only 1% of data to DAQ • Fixed Latency, NOT Serdes, 800Mb/s • Prototype built on MCH from NAT, but not required. • Vendor independent Greg Iles, Imperial College

  14. MicroTCA Disadvantages • Board thickness = 1.6mm (limited by edge connector) • New Harting connector = 2.0mm • Limited number of backplane I/O • 8 bidirectional I/O spare • Depending on application may be able to increase to 16 • Not suitable for Full mesh backplane • PCIe v Telecom clocks • PCIe system stole the AMC-Clk3 used in redundant telecom systems • Required because PCIe usually uses a spread spectrum clock distributed to all cards • PCIe can optionally operate without a “Fabric” clock • No Rear Transition module • Not convinced this is an issue for us Greg Iles, Imperial College

  15. Communication PHY • Protocol format for register read/write capability over large latency communication medium • i.e GbE in this case • Single data packet, multiple transactions • UDP/IP can be implemented in VHDL • Two versions already exist • TCP/IP usually implemented with processor • PowerPC hardcore • MicroBlaze soft-core • If hardware accelerated > 500Mb/s EMAC UDP, or TCP Transaction Engine I2C Core GTX Core DAQ Core Greg Iles, Imperial College

  16. Hardware controller PC separates the Control LAN and the User code from the Hardware LAN and the devices Unlike current TS architecture, all network traffic hidden from end user Made possible by common interface layer within the firmware and mirrored within the software Control LAN Fabric Hardware LAN Fabric User code User code User code Software: Architecture Single Multicore host Network Interface Multiplexer layer Transport Adapter Kernel Async. IO services Greg Iles, Imperial College

  17. Gaining momentum... • Jeremy Manns & Erich Frahm, Minnesota University • Specified protocol for large latency communication bus • i.e. Ethernet in this case • Single data packet, Multiple transactions • Provided UDP Verilog Core • Rob Frazier & Dave Newbold, Bristol University • Provide HAL to access cards • Online software control • Trek (?) hardware accelerated processor solution for TCP/IP • Wim Beaumont, Universiteit Antwerpen • TCP/IP, IPMI, Card infrastructure • Wesley Smith & Tom Gorski • TCP/IP with Xilinx microblaze Greg Iles, Imperial College

  18. Outstanding Issues • Crate cooling • Front-back, Top-bottom or flexible • Rack cooling • Vertical air flow with heat exchangers inside rack • Heat exchanger inside rear door • Some subtleties about Telecom/PCIe clock distribution • Rear transition modules? Greg Iles, Imperial College

  19. Physics xTCA working group Interested parties seem to be DESY and Schroff Rear Transition Module Greg Iles, Imperial College

  20. Physics Profile for comparison Greg Iles, Imperial College

  21. http://www.42u.com/42u-rack-cooling.htm Cooling • More efficient to cool a small amount of hot air close to the heat source, rather than large volume of luke warm air • Use hot/cold aisle containment to improve air flow and efficiency • As rack power dissipation has gone up cooling has moved closer to rack • We have this in CMS! • But industry seems keen to separate cooling/server racks • e.g. “In Row” and “In Rack” cooling In row cooling In rack cooling Greg Iles, Imperial College

  22. Questions ? DRAFT document on MicroTCA in physics: http://giles.web.cern.ch/giles/projects/slhc/The_CMS_uTCA_Crate_v0.6.pdf

  23. Hardware: MINI-T5 QFSPs 40 Gb/s bidirectional MicroController 2x40 LVDS 800Mb/s XC5VT150/240T SNAP12 / PPOD 120/60 Gb/s primary input/output http://giles.web.cern.ch/GILES/projects/slhc/slhc.html Greg Iles, Imperial College

More Related