FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group

1. FELIX DesignUpgrades of detector readout meeting9 June 2014Lorne Levinson, for the FELIX group Upgrades of detector readout meeting, 9 July 2014

2. Guiding principles • Goal: a GBT to LAN “switch” with as little awareness of detector specifics as possible • No user code in FELIX (so far): configuration params for options • Separate data movement from data processing • Configurable data paths from E-link to network endpoint • Scalable at Front end, Back end, and switch levels • Do as much as possible in software • COTS FPGA PCIe board • But… see talk on TTC clock recovery by Jinlong Zhang • Readout, so latency is not important, but provide low latency GBT-to-GBT links • Future proof by following COTS market, separately upgradeable components: GBT, FPGA PCIe board, serve PC, network cards • Routing is for E-links; they are the logical connections Upgrades of detector readout meeting, 9 July 2014

3. Virtex-7 PCIe board From Global HiTech, PCIe gen 3, 8 lanes Virtex X690T 2x12 bidir CXP connectors, FMC Upgrades of detector readout meeting, 9 July 2014

4. Off- detector architecture with FELIX “ROD” Functions can now be separated, physically, or just logically. “ROD” = Event fragment builder/processor for ROS Upgrades of detector readout meeting, 9 July 2014

5. Upgrades of detector readout meeting, 9 July 2014

6. Routing • Configured via control port on main or separate network • Simple/fixed:FELIX port # / E-link-range → IPaddress / portor → Infiniband local identifier / queue pair • E-link packet protocol: • Allows several logical streams to use the same E-link  header with: stream-id + packet len (if not fixed length packets)FELIX port # / stream-id → IPaddress / port… • Cloning, broadcasts / multicasts possible, in either direction • Reliable transport from FELIX to end-point via TCPIP • Configurable Quality of Service per logical connection • Less-than-best-effort for event data for monitoring Upgrades of detector readout meeting, 9 July 2014

7. Use cases – 1 • Receive data from an E-link or bonded group of E-links (a channel) and route it to one or more network endpoints via TCP or UDP. Bits must be arranged into correct order. • Optional 8b/10b or HDLC checking and/or decoding • Optionally verify checksum • Packet boundary identification either by SOP/EOP or length, optionally there can be no packet boundary, just a byte stream. • If a maximum packet length is configured, data arriving after the maximum is discarded and this error is marked in the packet status word. • If streams are defined for this channel packets are routed according to the stream. The stream ID may contain an extension filled by the detector with part of the Level-1 ID or the Trigger Type for advanced event routing. • Sampling can be requested Upgrades of detector readout meeting, 9 July 2014

8. Upgrades of detector readout meeting, 9 July 2014

9. Use cases – 2 • Receive BUSY from an E-link command, internally from FELIX itself, or from the network. Then route it to a network endpoint or to a standard ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent to the TTC system. • Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast them to the requested E-links • The BC clock is supplied to the GBT because the FELIX's GBT link clock is the BC clock supplied by the TTC system to FELIX. • Three options: 2-bit, 4-bit, 8-bit as described in the datasheet. • Receive data from a TCP port and route it to one or more E-links • Optional 8b/10b or HDLC encoding • A specific multicast channel can broadcast packets to all out-going E-links with the "MC" attribute. Upgrades of detector readout meeting, 9 July 2014

10. Use case – 3 • Scheduled data transfer lists: setting and activating (list is per GBT, but E-links are enabled by a bit map.) Open Issue: Or, several lists per FELIX and each E-link configures to use a specific list. Each item in the list is qualified by a delay to wait from the previous item in the list. • Activation by TTC user bit • Activation by command received via TCP • Receive data from an E-link or bonded group of E-links (a channel) and route it to a Direct Output Low Latency link (probably a GBT). Typically only a particular stream in the channel would be so routed. Upgrades of detector readout meeting, 9 July 2014

11. Low latency path is between predefined GBT pairs. Uses “cut-through” Routing. Upgrades of detector readout meeting, 9 July 2014

12. FELIX: FW/SW interface Data interface PCIe low level packet format (fixed length) PCIe high level packets, spans low level packets Push/pull? Hints from RobinNP architecture Configuration parameter interface Need description of parameters that can generate:XML for OKS, VHDL for FPGA See (but needs to be updated): https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf Monitoring and statistics interface FPGA configuration interface IP core from HitechGlobal ~4k$ Upgrades of detector readout meeting, 9 July 2014

13. Upgrades of detector readout meeting, 9 July 2014

14. Upgrades of detector readout meeting, 9 July 2014

15. Backup Upgrades of detector readout meeting, 9 July 2014

16. Functionality of E-lane handlers From GBT: need to decode 8b/10 or HDLC data • Collect 2, 4, 8, 16 bits per 40MHz clock • Align using comma symbols • Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet • Add 1KB block headers and “chunk” trailers • Optionally verify checksum To GBT: need to encode 8b/10b or HDLC data • Read from FIFO • Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO • Chop into 2, 4, 8, 16 bits per 40MHz clock Probably run at 80MHz to get extra clocks for headers/trailers Upgrades of detector readout meeting, 9 July 2014

17. Clock recovery • The 2x40MHz clock comes from the TTC link, i.e. a recovered clock. • Probably OK for GTH’s in the same GTH block of 4 transceivers • ??? • Can it be distributed to the other blocks? • Does the recovered clock have sufficiently low jitter for the other GTH’s? • We should probably consider a TTC receiver and jitter remover on an FMC card. • rework the GLIB version so that the FMC is on top of the PCIe mother board instead of extending beyond the PCIe board boundary Upgrades of detector readout meeting, 9 July 2014

18. Error handling • Just beginning the considerable thought that needs to be done here. • Two issues: • Discovery of corrupted data • Surviving errors without out loosing event boundaries • This requires some effort on the front ends • Corruption within a GBT 80-bit word is detected by the GBT’s FEC • Hard to believe E-links can be mixed into each other, so expect errors within E-link channels: repeated clocks, lost clocks • If E-links use 8b/10b with packet framing codes, fixed length, or variable length with counts, then errors (not all) can be detected. • FELIX can flag these errors in the header it sends with a stream’s data Upgrades of detector readout meeting, 9 July 2014