SuperB FCTS/DAQ Protocol Proposal

1. SuperB FCTS/DAQ Protocol Proposal Gregory Dubois-Felsmann & Steffen LuitzSuperB workshop, Elba31 May 2008

2. Where things stand • Last detector workshop: • We agreed that a reasonable way to move forward was for us to propose a sketch of the protocol for event data movement between the front-end electronics and the DAQ system, for an assumption of a fully-triggered readout. • This is primarily intended to provoke concrete reactions and discussion from those who have to design and build the electronics. They will very likely know better than we will what the constraints are, and we expect that will lead to changes in the protocol. • Original goal: • Circulate a written draft in time for feedback and rethinking by Elba. • Reality: • Proposal discussed at a detector telecon.; writeup is in progress SuperB FCTS/DAQ protocol proposal

3. Reminder about basic assumptions • Previously established • Physics goals require an open trigger, e.g., similar to BaBar’s. • Level 1 Accept rate of ~100kTps (triggers per second) • Unless highly effective Level 1 Bhabha veto is developed • Expecting 50kTps Bhabhas... • Stretch goal: be able to handle 150kTps • Level 3 Accept rate of ~20kTps • Expect “Level 4” offline filter beyond that SuperB FCTS/DAQ protocol proposal

4. Start from BaBar • BaBar-Note-281 (v1.1) described the protocol for communication between the ROMs and the FEEs • The Conceptual Design document for the FCTS describes the extension of this protocol through the FCTS system. • More detail is available in the “FCTS Architecture” note • For now we are considering only the event data protocol, corresponding to the “run-time commands” in the BaBar protocol (viz. Section 3.1 of BaBar-Note-281) SuperB FCTS/DAQ protocol proposal

5. BaBar event data protocol • In normal triggered data acquisition running: • Signal from Level 1 trigger (GLT lines) goes to FCGM • If system is not “busy” (within 2.x us of previous command) or “full” (no FE buffers available) or “inhibited” (external signal), send L1Accept command through FCTS to DAQ crates & ROMs • ROM forwards L1Accept command to FEEs • FEEs capture a previously specified window of data into a buffer (in triggered, i.e., non-EMC, systems) • Sometime later, when available, ROM sends ReadEvent command to FEEs, which send back the earliest available buffer • This relies on a trigger delivery latency that is confined to a fixed jitter interval, with readout windows large enough to cover the uncertainty. SuperB FCTS/DAQ protocol proposal

6. Two choices • Basic requirement is no intrinsic per-L1Accept deadtime, and deadtime due to “full” designed to be at most ~1% at the nominal 100 kTps trigger rate. This can be achieved in two ways… SuperB FCTS/DAQ protocol proposal

7. Alternatives • BaBar-like fixed-latency model • Works (roughly) if it is possible to deliver L1Accepts at a minimum spacing equal to the shortest time interval by which the (assumed fully pipelined) trigger can distinguish consecutive events. • Essentially this means that there can’t be a meaningful limit on the minimum command spacing (100 ns ~ 1% deadtime). In practice, in almost any scenario this requires being able to handle overlapping readout windows. • You don’t have to be able to do this for an unlimited number of events, of course, just for bursts long enough that statistically you don’t get significant deadtime due to “full”. • Places very stringent requirements on FCTS-DAQ link • Variable latency with addressing by time, and queueing of triggers • In general requires additional pre-L1Accept “ring buffer”-type space, as closely-spaced triggers will effectively be delayed in transit. SuperB FCTS/DAQ protocol proposal

8. Buffering • We assume two levels of buffering in the FEEs, in both models: • A continuously-running ring buffer upstream of the L1Accepts, long enough for the maximum trigger latency, in either model. • In Model 1, its length can be essential equal to the trigger latency (plus some constant offset) • In Model 2, it needs to be longer than this by enough to handle 99% of anticipated trigger bursts. We need to do modeling to be quantitative about this, but we anticipate that the answer will be O(10us) of additional capacity (i.e., roughly a doubling). • A post-L1Accept buffer. This would likely be constructed as a number of fixed-size slots as in BaBar’s design. The number of L1Accept slots required needs to be determined by modeling, but is very likely to be substantially more than the four in the BaBar design. • The driving parameters are that events must be able to be acquired about ten times faster than in BaBar, but the actual readout probably cannot be comparably faster (link speeds are only 2x faster, and we probably cannot afford to have significantly more ROMs). • The amount of such buffering needed would be somewhat larger in Model 1. SuperB FCTS/DAQ protocol proposal

9. Buffering II • Overlapping readout windows need to be supported • This has implications for the copy of event data from the ring buffer to the post-L1Accept buffer. • We propose that the protocol support copy by reference when windows overlap. This reduces the internal bandwidth required in the FEEs. • This requires the system to model an event as composed potentially of one or more by-reference segments followed by a by-value segment. At a minimum, the by-value segment of an event must be retained somewhere in the system until enough time has passed that a future event cannot need any part of it. This could be done either in the FEE or in the ROM, trading off complexity in the FEE against complexity in the FCTS protocol and the ROM. • We expect to propose one possible implementation as a reference point but describe alternatives as well. SuperB FCTS/DAQ protocol proposal

10. Command protocol • BaBar • ROM-to-FEE commands are 12 bits: a 0, a 1, a 5-bit command code, and a 5-bit trigger tag (sequence number). At 60 MHz, this takes 192 ns to transmit. • FCTS-to-ROM commands are 104 bits: the full 56-bit 60MHz clock counter (the unique event key), the post-FCTS state of the 32 trigger bits, the 12-bit ROM-to-FEE command, and four flag bits. These take ~1.75 us to transmit, a significant fraction of the minimum command spacing. SuperB FCTS/DAQ protocol proposal

11. Command protocols for SuperB Model 1… • The BaBar ROM-to-FEE command content is probably OK. • The BaBar ROM-to-FEE command timing is barely compatible with Model 1 at the same clock speeds. • Ideally, for 1% deadtime a 100ns command interval would be needed. Somewhat longer intervals could be acceptable in several scenarios: • If the trigger cannot generate separate trigger decisions that close together, then they don’t need to be processed, but the longer this interval becomes, the more necessary it becomes for the trigger itself to be able to handle overlapping events and make appropriate decisions (e.g., not vetoing an time interval that contains both a Bhabha and a physics event). • If triggers must be delayed in transit because of a somewhat longer command interval, this can be OK as long as the accumulated delay in the maximum burst that needs to be supported (from modeling) is compatible with the trigger jitter specification. SuperB FCTS/DAQ protocol proposal

12. Command protocols for SuperB Model 1… • The BaBar FCTS-to-ROM command timing is completely unacceptable. • It would lead to intolerable trigger delivery delays. • The command word length could be somewhat shortened. Two possibilities: • The post-FCTS-decision trigger bits could be treated as event data, with the FCTS read out as an additional detector system. (This would preclude the ROMs’ making FEX or other decisions based on trigger content.) • The timestamp could probably be shortened from 56 bits to 40-45 bits by treating it as relative within each data run. This would require a run identifier to become part of the unique event key. • These measures are probably insufficient by themselves. The command delivery link would still need to be made several times faster. • This in turn might preclude combining commands with clock distribution in the FCTS. • If these paths are separated, the ROMs would likely have to be able to resync commands with the clock in order for a BaBar-like event build and out-of-sync detection scheme to work. • We are still thinking this through. SuperB FCTS/DAQ protocol proposal

13. Command protocols for SuperB Model 2… • The ROM-to-FEE command word needs to be extended to include a ring buffer address field. • A length field may also be needed if the resolution of overlapping events is made a responsibility of the FCTS and the ROMs. • The time resolution of the addressing will need to be somewhere between the system clock period (e.g., 16 ns) and the minimum useful time resolution of the trigger (perhaps 125 ns). • The ring buffer in Model 2 needs to be somewhat longer than the intrinsic trigger latency, to accommodate queued triggers. Pending detailed modeling, we are guessing that a buffer depth of several tens of us should be adequate. • This means that the address field should be in the range of 8-12 bits. (A length field, if required, could be 4-8 bits.) • In Model 2, the resulting increase in command transmission time is almost irrelevant: it just adds somewhat to the required ring buffer depth. SuperB FCTS/DAQ protocol proposal

14. Command protocols for SuperB Model 2… • The FCTS-to-ROM command word needs to be extended by at least the address field. • If the resolution of overlapping events is made a responsibility of the FCTS and the ROMs, the command word would need to be further extended to include a set of descriptors for the multiple segments of an event with overlap. • Each descriptor would probably be both an address and a length. • Here again slower link speeds just translate into additional ring buffer space required. • Even 2-3 us command transmission time (e.g., if the command word is much more complex than for BaBar) could be accommodated. SuperB FCTS/DAQ protocol proposal

15. Conclusion & proposal • Model 2 appears to us to be significantly more attractive, and provides additional flexibility in DAQ that we think is advisable. • It appears to significantly loosen the requirements on the performance of the FCTS and ROM-to-FEE command links. • The corresponding cost is the increased complexity of the FEE needed to support an addressable - and significantly larger - ring buffer. • We are proposing this model - that is, a model with time-addressable pre-L1Accept buffering in the FE electronics. Next steps: • Define the time addressing protocol and buffer depth. • Specify a model for overlapping-event readout. • Write this up. SuperB FCTS/DAQ protocol proposal

16. Requests to subsystems • Responses from two subsystem groups so far. • Thank you! • Additional question (or clarification): • Please include in your presentation of channel requirements (counts, bit depths, digitization rates, etc.) the readout window that is required by the system’s intrinsic signal distribution, ignoring trigger jitter. SuperB FCTS/DAQ protocol proposal