1 / 16

CRU Firmware Overview

CRU Firmware Overview. Erno DAVID, Tivadar KISS Wigner Research Center for Physics (HU). 15 March, 2016. Content. General view of the CRU Firmware Different communication forms between the CRU and the FEs The TTS Link and the GBT Links LHC Clock distribution Slow Control User Interfaces

evelyncain
Download Presentation

CRU Firmware Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CRU Firmware Overview Erno DAVID, Tivadar KISSWigner Research Center for Physics (HU) 15 March, 2016

  2. Content • General view of the CRU Firmware • Different communication forms between the CRU and the FEs • The TTS Link and the GBT Links • LHC Clock distribution • Slow Control • User Interfaces • Firmware Development Environment 2

  3. CRU Block Diagram CTP FLP Server CRU Common CRU FW/SW Components provided by the CRU team TTS LTU Extensible by the Detector teams RAM Detector Data Frames GBT User (detector specific) Logic PCIe Developed by the Detector and DCS teams FEE Detector Control Frames Developed by the Detector and O2 teams FEE CPU FLP DAQ Software to EPN Servers NIC GBTx, SCA, CRU Ctrl. FLP DCS Software NIC to DCS Servers

  4. Different Communication Forms Between the CRU and the FEs Trigger and Timing CRU 40 MHz LHC Clock + 188 bit data (Orbit, BC, …) Data LTU SFP+ TTS CTP Busy DCS Busy FEE 40 MHz LHC Clock + 80 bit trigger (Orbit, BC, …) GBT LatOpt. Tx (24/36/48 link) 1 GBTx (ASIC) ASIC orFPGA 40 MHz LHC Clock 1 1 or 40 MHz x 80 bit DCS . . . Trigger or DCS(up to 40 e-links) 3 3 User (detector specific) Logic 3 GBT Tx DCS 4 40 MHz x 2 bit Internal SC I2C, SPI, GPIO, JTAG 40 MHz x 2 bit External SC 80 Mb/s 5 GBT-SCA(ASIC) . . . 5 3 2 PCIe, x16, Gen3, 128 Gbps GBT Rx GBT Std. Rx (24/36/48 link) 3 40 MHz x 80 bit FEE Data DATA 2 40 MHz x 2 bit Internal SC FEE Data (up to 40 e-links) . . . 4 3 2 40 MHz x 2 bit External SC 5 4 5 DCS . . . • Main transactions: • Delivering the TTS information or Read-out Control from CRU to FE • Receiving the detector data in serial and parallel form from the FE • Communicating custom packets between the CRU and the FE FPGA in parallel form • Sending and receiving packets to and from GBTx internal register block • Sending and receiving packets to and from GBT-SCA ASIC 1 2 3 4 5

  5. TTS & GBT • TTS Link Downlink: • 10G PON, 9.6 Gbps, 8B/10B, 240-bit @ 40 MHz • Getting the trigger and throttling (drop-maps) information (up to 184-bit @ 40 MHz) • TTS Uplink: • 10G PON, 2.4 Gbps • Propagating back the busy information from CRU to the LTU • GBT Downlinks: • 24/36/48 4.8 Gbps GBT links with Forward Error Correction (FEC, 80-bit mode) • Deterministic latency • Distribute: LHC Clock, trigger information, slow control • GBT Uplinks: • 24/36/48 4.8 Gbps GBT links in 80-bit or in 112-bit (Wide Bus) mode • Deterministic latency • Distribute: Detector data, slow control 5

  6. LHC Clock Distribution (1) CRU Arria 10 FPGA TTS(ONU) 10G PON REFCLK120 MHz 10G PON Transceiver 240MHz 40 MHz 10G PON External PLL + Jitter Cleaner + Fanout Buffer 1 2 40 MHz 184 bit Busy 7 6 5 4 3 2 1 0 Phase Control PMA Reset 120 MHz 3 GBT Transmitters Bank 0 GBT TX Downlink Logic Bank 0 250 MHz GBT 80/112 bit 6 x 40 bit 512 bit 2400 MHz TX SCA ATX PLL 6 x 2 bit GBT_BANK_0_REFCLK 120 MHz TX GBTx 2 bit GBT Bank 0 – Link 0 .. 5 (Consist 6 adjacent bonded Arria 10 transceiver) 120 MHz GBT RX Uplink Logic Bank 0 250 MHz GBT Receivers Bank 0 80/112 bit 6 x 40 bit 512 bit 6 x RX SCA 2 bit The number of GBT Banks is equal to GBT Links / 6 RX GBTx 2 bit 6 GBT_BANK_7_REFCLK 120 MHz

  7. LHC Clock Distribution (2) CRU PON Frame 6x40 bits @ 40 MHz 9.6 PON Downlink LTU K28.5 K28.5 K28.5 payload K28.5 K28.5 K28.5 . . . PON Receiver 40 bit 40 bit 1 LHC clock (40 MHz) PON RX clock (240 MHz) PON TX clock (240 MHz) 2 Reconstructed LHC clock (40 MHz) GBT REF clock (120 MHz) 3 GBT TX clock (120 MHz) GBT Frame 3x40 bits @ 40 MHz 4.8 GBT Downlink GBTX H H H payload H H H GBT Transmitter • There will have one 240 MHz PON RX clock domain • There will up to eight 120 MHz GBT TX clock domain • The phase difference between the PON RX clock and the GBT TX clocks determines: • The distributed LHC clock phase • Trigger information delay Reconstructed LHC clock (40 MHz) 1 3

  8. TTC-PON’15 Arria 10 LHC Clock Reconstruction OLT ONU 120 MHz TTC-PON’15 Receiver TTC-PON’15 Transmitter refclk 40 MHz 240 MHz PLL refclk 1 2 1.a 2.b 2.a 40 MHz 240 MHz PLL 1.b TTC-PON’15 Frame Generator 240 MHz 40 bit PON Rx Clock Aligner 9.6 Gbps rst phase x bit payload frame locked 40 bit LHC Clock Aligner locked 3 bit pon word counter 1.c analogreset valid clockslip First implemented in Arria 10 Later will be moved to Kintex 7 184 bit 2.c Implemented in Arria 10 Source 40 MHz LHC Clock 1.a 240 MHz Tx Clock 1.b Reconstructed 40 MHz LHC Clock 2.a 1.c Source payload bits 240 MHz Rx Clock 2.b 2.c Reconstructed payload bits 8

  9. Identified Problems with the LHC Clock Distribution 9

  10. GBT Frames vs. Different CRU – FE Communication Forms 120 bit @ 40 MHz GBT Downlink (CRU -> FE) 4 bit 2 bit 2 bit 80 bit 32 bit GBT frames: User Data(GBT data channel) Header IC EC FEC(optional) (GBT ctrl ch) InternalControl:slow control access to GBTx control registers(e.g multiple detectors) ExternalControl:slow control access to FE electronics via GBT-SCA(I2C, SPI, MDIO, JTAG or GPIO)(e.g. multiple detectors) E-links: Multiple independent detector specific FE read-out control signals(e.g. TPC) E-links: 80-bit wide custom data packets to FE FPGA(e.g. ITS) E-links: 80-bit TTS informationsent continuously @ 40 MHz(e.g. TOF) 120 bit @ 40 MHz GBT Uplink (FE -> CRU) 32 bit 4 bit 2 bit 2 bit 80 bit GBT frames: IC User Data(GBT data channel) Header EC FEC(optional) (GBT ctrl ch) E-links: Independent serial raw data streams (frames) from multiple FE ASICs(e.g. MCH) ExternalControl:slow control access to FE electronics via GBT-SCA(I2C, SPI, MDIO, JTAG or GPIO)(e.g. multiple detectors) InternalControl:slow control access to GBTx control registers(e.g. multiple detectors) E-links: 112-bit Wide Bus mode raw data streams from multiple FE ASICs(e.g. TPC) E-links: 80-bit wide formatted data packets w/ CDH/SDH*from FE FPGA (e.g. TOF)

  11. Packet Transmission Example DATAVALID . . . 0 79 IDLE 0 0 SOP Length (optional) TTS Busy time GBT word -1 1 . . . GBT word 0 CDH Payload – segment 1 1 . . . . . . 1 GBT word Length-1 EOP Length Checksum End Flag 0 GBT word Length+0 0 SOP Length (optional) TTS Busy GBT word -1 . . . 1 GBT word 0 CDH Payload – segment 2 1 . . . . . . 1 GBT word Length-1 EOP Length Checksum End Flag 0 GBT word Length+0 IDLE 0 . . . 80-bit GBT words

  12. Slow Control • Main parts: • GBT-SCA – Main communication channel with the front-end • GBTx ASIC – GBT link configuration – front-end side • CRU own slow control • Onboard peripherals (MiniPODs, external PLL, …) • Internal firmware modules • Multiple masters: • SW – O2 • SW – DCS • FW – Safety module 12

  13. CRU Firmware Slow Control SoC Architecture FLP Server PCIe40 Board Linux CRU API Arria 10 FPGA O2 Software GBT-SCA User Logic TTS, GBT,…(core modules) [0..47] x 2 bit @ 40 MHz Custom Interface [0..47] x 2 bit @ 40 MHz QSYS PCIe Endpoint 0bar_x_masterAvalon-MM Master250 MHz x 32/64 bit DCS Software QSYS Interconnect (250 MHz x 32/64 bit) ??? Software GBTx ASIC I2C ? [0..47] x 2 bit @ 40 MHz [0..47] x 2 bit @ 40 MHz SFP+ Si 5338 MiniPOD 0 MiniPOD 7 Arria 10 Firmware Flash . . . 13

  14. CRU User Interfaces (1) CTP FLP Server CRU Common CRU FW/SW Components provided by the CRU team TTS LTU Extensible by the Detector teams RAM 1 Detector Data Frames GBT User (detector specific) Logic PCIe Developed by the Detector and DCS teams FEE 3 2 Detector Control Frames Developed by the Detector and O2 teams FEE 4 CPU FLP DAQ Software to EPN Servers NIC GBTx, SCA, CRU Ctrl. FLP DCS Software NIC to DCS Servers • Main User Interfaces: • TTS Downlink and Uplink • GBT Downlinks and Uplinks • PCIe Link • Slow Control Links: Avalon-MM Slave Interface, GBT-SCA firmware level interface 1 2 3 4

  15. CRU User Interfaces (2) • TTS Downlink (slave): Trigger information, stream like (no handshake), 184 payload bits, updated @ 40 MHz • TTS Uplink (master): Busy information, FIFO like interface • GBT Downlinks (master): 80+4 bit wide, stream like (no handshake), updated @ 40 MHz • GBT Uplinks (slave): 80/112+4 bit wide, stream like (no handshake), updated @ 40 MHz • PCIe Link: Bidirectional FIFO like interface, 250 MHz, 512-bit • Slow Control – Avalon-MM Slave: Register based 32-bit interface • Slow Control – GBT-SCA firmware level interface (master)

  16. CRU Firmware Development Environment 10gpon_eval gbtsca_eval pciedma_eval pcie40cru ├───core │ ├───firmware │ │ ├───scripts │ │ ├───sim │ │ └───src │ └───software │ ├───drivers │ └───tools ├───generic │ ├───firmware │ └───software ├───tpc ├───its ├───mft ├───trd ├───mch └───mid • Multiple independent evaluation projects. • They will be later integrated into the main CRU git repository. • Single central place which contains the core CRU framework features and the detector specific user logic codes. • It allows to rebuild all the CRU firmware variations from a single place. • Planned support for extensive user logic simulation / validation. • Buildable from command line, tested with build servers. • GitLab URL: https://gitlab.cern.ch/alice-cru/pcie40cru 16

More Related