1 MHz Readout with a Star Hybrid Overview

1. 1 MHz Readout with a Star HybridOverview Matt Warren (with lots of input from Tony Affolder, Francis Anghinolfi, Ingo Bloch, Richard Brenner, Gordon Crone, Sergio Diez Cornell, Philippe Farthouat, Bruce Gallop, Ashley Greenall, Ingrid Gregor, Alex Grillo, Carl Haber, Tim Jones, Paul Keener, Nikos Konstantinidis, Andreas Korn, Steve McMahon, Mitch Newcomer, Peter Phillips, Marcel Stanitzki, Tony Weidberg and more) 9 Sept 2014

2. 1MHz Star Introduction • Recent history: Baseline trigger rate requirements changed • Double that of the LoI • 1MHz L0 trigger, 400kHz L1, ~100kHz R3 = 500kHz Readout • To cope with this we “simply” doubled the BW allocations • 320Mb/s electrical links (was 160Mb) • 2 GBTs (or 1 lpGBT) per stave/petal-side (was 1GBT) • Also new: TDAQ require new designs to have a • Minimum L0 latency of 10μs • Minimum L1 latency of 60 μs • Trigger on successive beam crossings • Next request: Can we readout at L0? = 1MHz Readout • No need for regional readout – all data is available in the counting-room • Implies doubling BW again = 4 GBTs per stave-side (or 2 lpGBTs) • L1Track is also squeezing latency of R3 data readout Time for a re-think …

3. 1MHz Star Background: The LoI Hybrid • LoI hybrid is optimised for relatively low occupancy • Each ABC generates complete, self-describing data packets • Packets are fixed length to reduce bit count and improve predictability • Each packet comprises ~60% hit data – the rest being headers, L0ID etc. • Slightly wasteful of BW but robust and fits with 1 GBT per stave-side • ABCs connect to a HCC, HCCs to a GBT • ABCs are connected in 2 chains of 5 • Each chain has a Data line and an Xoff line at each end • 4 lines/chain = 8 per hybrid • L1 and R3 “triggers” are actually short messages • Contain the L0ID of the desired event • Scaling this to 1MHz readout exposes bottlenecks • Prompting a look at an alternative topology: the Star Hybrid

4. The New(comer) Idea: Star Hybrid • Each ABC has a dedicated line to the HCC • No XOff lines – the HCC requests data from ASICs when needed • L0 trigger unchanged – passed through the HCC to ABCs • But now the L0-BCID is stored (and queued) on HCC • On the ABC, the L0 still moves data from pipeline to buffer as before • Allows for both L1 and L0 readout options, with R3 option too • L1 readout: • L1 triggers are queued on the HCC • Only sent to ABCs when HCC is ready • L0 readout: • HCC monitors L0s as they pass • Generates “L1” requests and writes them to L1 request queue • NOTE: Without L1/R3, the L1 request simplifies to a “Next” command • R3 data readout (in parallel with the above): • handled like L1, with separate queue • Request sent to ABC before any queued L1s = lower latency 1MHz Star

5. 1MHz Star Star Hybrid (2) • ABCs send only hits (no meta data needed) • No ChipID – determined from physical connection on HCC • L0ID and BCID are already known to HCC • FE-ASIC outputs data at much higher BW than HCC • Allows “fast fill” of HCC to overcome peaks in occupancy • Packets are built on the HCC • One event per packet • 1 set of event identifiers (L0ID, BCID) • Can have parity, error correction etc • Packet are sent in order – helps off-detector • Better fit with FELIX • Buffering is done on the ABCs – the HCC only needs a few levels • 1) Data receiving from ABCs, 2) Packet build, 3) Sending • Packets are built in order • Incrementing L0IDs are easy to track in logic • Each ROD link deals with 1 event at a time • Lower timeouts for lost data – the next event will have a new BCID/L0ID

6. 1MHz Star Redundant Diagram? HCC ABC L0 L1 FIFO ABC Send L1 Stave Bus Control … Half-full FIFO ABC Data

7. Simulation Results, BW estimate • For inner barrel hybrid: 12.5 clusters per event = 13 • Where cluster = Strip number + bitmap of 3 adjacent strips • Each cluster is 12b strip address + 3b adj-hit-map = 15b • Note: Wide spread of number of clusters per event • Fixed length packets not efficient • HCC packet data: BW ~250Mb/s 1MHz Star

8. 1MHz Star Implications of Star Hybrid • More links on hybrid • +2 for barrel, + 4 more for worst case EC hybrid • New hybrid layout (extra circuit layer for extra links) • Slight increase in material • New HCC design • More complex • Updated ABC design • But not too radical • 320Mb/s link for each hybrid = ~9Gb/stave • 2xGBT or 1xLpGBT • More material at EoS, or more risk • Stave/Petal tapes unchanged • But relies on the 320Mb links working (only fully tested to 160Mb) • Will need modified EoS for 2 GBT solution

9. 1MHz Star Comments/Conclusions • The HCC is the natural place to put parallel processing of hybrid data • We have a means of reading out at 1MHz, but needs more simulation • The ABC need not be completely redesigned, but still significant work • Changes to buffer sizes, data format, trigger queues • Final tweaks can be made to the HCC later than ABC • Far fewer in the system, can be (prefabricated last • Even without 1MHz readout, changes to the architecture are beneficial • More efficient data packing = ~40% saving in BW • Data leaves hybrid in order - Much easier for off-detector • Regional Data (R3-data) latency can be reduced • Why keep L1/R3? • Flexibility - what if 1MHz isn’t possible in the end? • Latency concerns: 1MHz readout is 1us/event average, peaking at 40MHz. A burst of >1MHz triggers could delay data too long. This needs simulation! • A document is evolving here: http://www.hep.ucl.ac.uk/~warren/upgrade/ITk_Strips_at_1MHz.pdf

10. 1MHz Star Topics for discussion • Is Star is the way to go? • Should we aim for 1 MHz L0 readout? • Do we keep R3+L1 functionality?

11. 1MHz Star • Backup

12. 1MHz Star More thoughts … • Data from ABC may need to be capped per request i.e. When, say, >10 clusters the ABC sends it’s data in 2 blocks, each with it’s own L1 request • Will allow optimal buffer sizes • Needs special control protocol and trigger • i.e. multiple L1 requests for same L0ID • Similarly HCC should have capped packet len • Efficiencies in HCC packet builder • Allows for more efficient off-detector buffering/handling • May need a mode where ABC do send L0ID and/or BCID • For timing in and test purposes. • Stand-alone ABC • Explore 640Mb out of HCC • 2 HCCs per link/redundancy – i.e. “neighbour input” • Fast tapes option