GBT in FPGAs

1. Sophie Baron, Radhard Link Status Meeting System View Implementation Optimization Starter Kit People GBT in FPGAs 3/11/2010

2. Sophie Baron, Radhard Link Status Meeting SYSTEM VIEW

3. Sophie Baron, Radhard Link Status Meeting SYSTEM VIEW

4. Common need for all users • An FPGA will be needed to communicate with FE boards • i.e. with as many GBT links as possible to concentrate upstream data (typically 48x 5Gb/s links max in latest FPGAs) • The GBT protocol is too specific to be available in standard FPGA transceivers IPs • A custom firmware core will be required • Besides… • This core would allow any potential GBT user to play with the protocol and the transmission links far before having the chip on hand Sophie Baron, Radhard Link Status Meeting GBT-FPGA

5. A « GBT-FPGA » group was created in 2009 with early users of a GBT protocol in an FPGA (CERN, IN2P3 Marseille) • Objectives: • Build a complete SERDES for GBT in FPGA • To provide FPGA to GBT communication • To have a VHDL model of the GBT chip • Use generic VHDL as much as possible (limit the vendor specific cores to transceivers, PLL, RAM) • Evaluate it with both Altera & Xilinx devices • Release it as a firmware « Starter Kit » supported by CERN (one version per main device type) • Investigate in optimization techniques for resources and latency Sophie Baron, Radhard Link Status Meeting GBT-FPGA

6. Sophie Baron, Radhard Link Status Meeting Implementation Implementing the GBT data transmission protocol in FPGAs, F. Marin et Al, TWEPP09

7. Sources stored on SVN as a working basis for the development team Most of the files are generic VHDL A few are devicespecific Early version testedduringspring 2009 Sophie Baron, Radhard Link Status Meeting implementation Implementing the GBT data transmission protocol in FPGAs, F. Marin et Al, TWEPP09

8. Sophie Baron, Radhard Link Status Meeting PERFORMANCES Implementing the GBT data transmission protocol in FPGAs, F. Marin et Al, TWEPP09 VIRTEX6LXT Optimizinglatency in Xilinx FPGA Implementations of the GBT, S. Muschter et Al, TWEPP10

9. Resources occupancy • Typical values for 16-transceivers FPGAs (medium size): • StratixIIGx EP2SGX90 with 16/16 transceivers used: 69% of the logic • Virtex 5 XC5VFX100T with 13/16 transceivers used: 93% • Mostly due to the decoder occupancy • Better resources usage with new device generations: • Virtex6 V6LX240T with 12/24 transceivers used: 17% • Very high latency due to many synchronization stages • 21 clock cycles for a loopback • 9 for TX • 8 for RX • 1-4 for the GTX depending on the instantiation & device Sophie Baron, Radhard Link Status Meeting LIMITATIONS of the early version Implementing the GBT data transmission protocol in FPGAs, F. Marin et Al, TWEPP09

10. Decoder sharing • Overclocking the decoder and sharing it between several receivers • Tested for Virtex5 and Stratix II. Max factor:1/3. • To be evaluated for SIV and V6: probably better ratio could be reached • Other ideas: • Pipelining the decoder to speed it up • Goes against latency optimization • Removing decoder for radiation free areas • No error correction Sophie Baron, Radhard Link Status Meeting Resources Optimization (2009) Implementing the GBT data transmission protocol in FPGAs, F. Marin et Al, TWEPP09

11. Study made by S. Muschter in 2010 and presented at TWEPP 2010 • Framework: low & fixed latency required for trigger data transmission to trigger processor • Hardware: Virtex6 LXT with GTX transceivers 6.6Gb/s • Latency of the early version • Low latency was not considered as a requirement on the first version • Tx: 9, Rx: 8 clock cycles • Latency of the optimized version • RAM flavor: Tx:3, Rx:4. • Register flavor: Tx:2, Rx:3. • Deterministic latency checked in loopback with a common 40MHz Reference on Rx and Tx Sophie Baron, Radhard Link Status Meeting LATENCY OPTIMIZATION (2010) Optimizinglatency in Xilinx FPGA Implementations of the GBT, S. Muschter et Al, TWEPP10

12. A starter kit is available since February 2010 on the GBT-FPGA sharepoint site: https://espace.cern.ch/GBT-Project/GBT-FPGA/default.aspx • Documentation • VHDL files organised as a tree including • Generic VHDL sources • Device specific entities (StratixII & Virtex5) • Project for each device type with a library calling the right files • Resource optimization options • A new version with the latency optimization will be available before end of 2010 Sophie Baron, Radhard Link Status Meeting Starter kit release

13. 2010 : • Release latency optimization • Early 2011: • Test resources optimization with SIV & V6 • Release versions for SIV & V6 • 2011: • Add e-links • Investigate on further optimization schemes Sophie Baron, Radhard Link Status Meeting FUTHER PLANS

14. Users • 30 registered users from all over the world (most users from collaborating institutes) • LHC experiments, but also CLIC, PANDA, GBT • Very active users are now part of the development team (Steffen Muschter, Stockholm) • Design Team • CsabaSoos, CERN (Virtex5 & Virtex6) • Jean Pierre Cachemiche, IN2P3 Marseille (Stratix II and IV) • Steffen Muschter, Stockholm University, Atlas Tilecal (Virtex6 and latencyoptimization) • Sophie Baron (Stratix II and IV, resourceoptimization, general support) • Frederic Marin (whopresented the GBT-FPGA paperat TWEPP09) left IN2P3 Marseille 6 monthsago Sophie Baron, Radhard Link Status Meeting PEOPLE

15. GBT Project sharepoint site: https://espace.cern.ch/GBT-Project/default.aspx GBT-FPGA sharepoint site: https://espace.cern.ch/GBT-Project/GBT-FPGA/default.aspx GBT System Integration, J. Christiansen: https://espace.cern.ch/GBT-Project/GBTX/Specifications/GBT_system_integration_v1.3.pdf Implementing the GBT data transmission protocol in FPGA (F. Marin, TWEPP2009): http://indico.cern.ch/contributionDisplay.py?sessionId=35&contribId=78&confId=49682 Optimizing latency in Xilinx FPGA implementations of the GBT (S. Muschter, TWEPP10): http://indico.cern.ch/contributionDisplay.py?sessionId=31&contribId=61&confId=83060 Sophie Baron, Radhard Link Status Meeting Useful links

16. Sophie Baron, Radhard Link Status Meeting Thankyou