1 / 28

CBM FEE/DAQ/Trigger

CBM FEE/DAQ/Trigger. System and Network Architecture. Walter F.J. Müller , GSI 2 nd FutureDAQ workshop, September 9, 2004. Some Remarks. The following does not explore the full design space It looks into one scenario, driven by the Ansatz

gin
Download Presentation

CBM FEE/DAQ/Trigger

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CBM FEE/DAQ/Trigger System and Network Architecture Walter F.J. Müller, GSI 2nd FutureDAQ workshop, September 9, 2004

  2. Some Remarks • The following does not explore the full design space • It looks into one scenario, driven by the Ansatz do (almost) all processing done after the build stage • Other scenarios are possible and should be investigated, putting more emphasis on: • do all processing as early as possible • transfer data only then necessary • All specific 'implementations' are just examples for illustration • All stated numbers are still rough estimates CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  3. FEE CNet PNet A Network oriented System View Front-endElectronics ConcentratorNetworks BuildNetwork BNet ProcessorNetworks ComputeResources High LevelNetwork HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  4. ... with some numbers ~ 50000FEE chips FEE CNet ~ 1 TB/Sec BW~ 1000 links BNet PNet ~ 100sub farms ~ 100 nodesper sub farm HNet Output BW~ 10 GB/sec to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  5. ... with some questions What is theFEE Link Protocol ? FEE Where is thetime distribution ? CNet Interfacing ofCNet to BNet ? BNet Interfacing ofBNet to PNet ? PNet Interfacing ofHNet to PNet ? HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  6. FEE CNet Buffer Node atCNet – BNet Interface Buffer Node atBNet – PNet Interface PNet Processor Node atPNet – HNet Interface ... and now decoupling the Networks BNet HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  7. Proposal (V.Lindenstruth): handle all three interfaces with one physical serial link use an optical interface already at the FEE chip level + no noise from data lines + low chip pin count this requires SerDes cores on FEE chips low-cost fiber couplings this implies clock/time is distributed on the uplinks of the CNet FEE Interface 3 logical interfaces FEE OASEunderdevelopment Hit Data(out only) Clock and Time(in only) see talk by K.H. Brenner Control(bidirectional) CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  8. FEE and CNet Setup FEE FEE FEE FEE FEE FEE FEE FEE Up link Down link Tree or Bayanstructure clocktimerequestflow ctrl dataresponse Depending on multiplex ratio single- or multi-level CNet – BNetInterface CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  9. Clock/Time Distribution • Definition of 'Time distribution': • provides global state transitions with clock cycle precise latency • Consequences for serial FEE links and CNet: • bit clock cycle precise transmission of time messagesover links and through CNet concentrator switches • low jitter clock recover required • FEE link and CNet will likely use custom protocol • CNet-Bnet buffer is natural injection point for global time messages • otherwise BNet inherits CNet requirements • this implies yet another net, the TNet to distribute the global time messages to the CNet-Bnet buffers CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  10. FEE CNet PNet ... so now with 5 Networks TNet BNet TNetInterface Connect toBNet or a PNet HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  11. FEE CNet PNet Network Characteristics Data PushDatagram'serrors markedbut not recovered TNet BNet Request/Responseand Data PushTransactionserrors recovered HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  12. FEE Data – Basics • FEE data is a stream of hits and epoch marker messages • A hit message contains in simplest case • chip Id (unique at least in local CNet) ~ 8...12 bit • channel Id ~ 4... 9 bit • time stamp (relative to epoch) ~14...20 bit • amplitude ~ 8...12 bit • An epoch marker message contains in simplest case • epoch number (at least some LSB's) • → FEE outputs short messages CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  13. FEE Data – Clusters • In most subsystems a particle causes correlated hits in physically neighboring detector cells (STS, TRD, ECAL) • Depending on detector subsystem • the cluster pattern is 1d or 2d • contained in one FEE chip or not • STS-MAPS: 2d contained • STS-Strip: 1d mostly contained • RICH single photon hits on PMT's → no clusters • TRD 1d mostly contained to 2d often uncontained depending on pad geometry (varies inside→outside) • RPC t.b.d. • ECAL 2d mostly uncontained • Doing (partial) cluster finding reduces data volume by ~50% • price to pay: in some cases (e.g. ECAL) varying length messages CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  14. FEE Development – Status • The TRD has 500k – 1000k channels → ASIC inevitable • Self-triggered operation at high rate prevents using most existing ASIC's → some new development inevitable • Approach: • avoid too specialized chips • aim at a small family of highly related chips • → coordinate development • → select common technology base and building blocks • First CBM FEE coordination meeting • Date/Place: September 24-26 at CERN • Scope: (MAPS), Si-Strip, RICH, TRD, RPC, ECAL • Aim: Common architecture for RICH, TRD, ECAL Common building blocks also for RPC, Si-Strip visitors welcome CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  15. FEE Data – Summary • Hit messages may be variable length • Hit messages are short (5-10 bytes, small upper bound) • Epoch length is a free parameter • reasonable values are around 10 μsec (see also MAPS frame time...) • CNet switch functionality quite non-standard • bit clock cycle precise time message forwarding • epoch message handling (they act as time barrier and heart beat) • data stream merge semantics • FEE link and CNet switch functionality under discussion • FEE and CNet components will be co-developed CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  16. FEE CNet PNet Switch focus CNet → PNet TNet BNet HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  17. A sub-farm is a collection of compute resources connected with a PNet Compute resources are programmable logic (FPGA) processors Likely choice for the processors are high performance SoC components CPUs, MEM, high speed interconnect on one chip optimized for low W/GFlop and high packing density see QCDOC, Blue Gene, STI cell, .... Best choice for PNet is to use 'build-in' serial links and connect them with switches PCIe-AS is a plausible candidate for a commonly used serial interconnect and thus for PNet Structure of a Sub-farm I see talk by J. Gläß from BNet PNet to HNet see talk by E. Denes CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  18. High Performance SoC Systems see also talk byLars Schmitt • Get some inspiration from Supercomputing • Have a look at IBM Blue Gene/L technology • many interesting talks in the workshop BlueGene/L: Applications, Architecture and Software Sparks, Nevada, October 14–15, 2003 organized by Lawerence Livermoore Lab http://www.llnl.gov/asci/platforms/bluegene/bluegene_index.html • Note 1: BlueGene and QCDOC use similar technology • Note 2: STI Cell processor seem more innovative, but very little information has been published so far CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  19. A plausible scenario for the low level compute farm ~100 sub-farms each with ~100 compute resources a sub-farm has ~10 boards, each with ~10 resources a sub-farm fits in one crate use crate providing serial backplane fabric look into PICMG 2.17, 2.20 and VITA 41.x (VXS) Consequences only chip-2-chip and board-2-board links in PNet thus only short distance (<1m) communication this is what PCIe-AS is developed for .... take VITA 41.4 VXS backplane as a model Structure of a Sub-farm II from BNet PNet to HNet CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  20. 10 node boards(different types) double star fabric 10 Gbps links 40 GB/sec BW feasible TODAY !! A Sub-farm Scenario 8 portswitches 8 portswitches FabricBoard Node Board from BNet 10 Gbpslinks 10 Gbpslinks to HNet CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  21. A Sub-farm Scenario II MultiGig RTconnectors6.4 Gpbs/pin MultiGig RTconnectors6.4 Gpbs/pin This is VME++VITA 41.x • Core pieces are already here today • can only improve... • Missing: FPGA and SoC with PCIe • wait and see how market moves • Essential: Power consumption • 2.5 Gbps SerDes now 100-200 mW • 160 Gbps switch now 6.5 W (130 nm) • PLX PEX8532 or BCM5675 • in 2010: see what 45nm will do 8 port, 32 lanePCIe-AS switch160 Gbps bandwidthPLX PEX8532 10+2 slot VXS backplaneELMA 101VXSD712 CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  22. FEE CNet PNet Switch focus PNet → BNet TNet BNet HNet to high level computing CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  23. Building Network I see talk by S. Linev • What is being build here ? • Choices are: • time intervals • event association will be handled in PNet compute resources • simple and straightforward solution • variable length time slices (e.g. 2-16 epochs) for load equalization • all raw data goes through BNet • event intervals • event detection/association handled in CNet-BNet buffer • Note: MAPS data will always requires time interval based handling • Allows suppression of • incoherent background • peripheral events (e.g. b dependent scale down) • potentially significant reduction of BNet traffic CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  24. Building Network II • There are ~1000 links out of the CNets • BW conservation requires ~1000 links into the PNets • Do we need a big 1000 x 1000 switch ? • Answer: NO • very specific, predictable, and constant traffic pattern • traffic not random, can be shaped (e.g. via tags send over TNet) • switch can be easily factorized: • Assume ~100 PNets, each connect with ~10 links to BNet • This allows to factorize BNet into ~10 independent subnets • Each BNet subnet, now a ~100 x ~100 switch, can be build from two layers of ~10 x ~10 switches • Bottom line: • ~200 switches of ~10 x ~10 are sufficient (each a now a single chip !!) • many other factorizations possible CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  25. A possible Scheme 128 sub-farms 8 input ports per sub-farm 8 BNet subnets each with 128 input ports 128 output ports BNet Example I BNet PNet CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  26. A possible Scheme each of 8 BNets build from 2 layers of 12x12 switches BNet Example II BNet PNet CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  27. Summary see talk by P. Haspel see also talk by K. Korcyl • 5 different networks with very different characteristics • CNet • medium distance, short messages, special requirements • connects custom components (FEE ASICs) • TNet • broadcast time (and tags), special requirements • BNet • naturally large messages, Rack-2-Rack • PNet • short distance, most efficient if already 'build-in' • connects standard components (FPGA, SoCs) • HNet • general purpose, to rest of world Custom Custom quite open...10GE Cat6PCIe-AS,..... PCIe-AS Ethernet CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

  28. Summary • 5 different networks with very different characteristics • CNet • medium distance, short messages, special requirements • connects custom components (FEE ASICs) • TNet • broadcast time (and tags), special requirements • BNet • naturally large messages, Rack-2-Rack • PNet • short distance, most efficient if already 'build-in' • connects standard components (FPGA, SoCs) • HNet • general purpose, to rest of world FEE Interfaces and CNet will be co-developed. FEE task force being formed Decision on R&Dfocus soon Decision on R&Dfocus soon Custom Potentially build with CNet components Custom Probably uncritical quite open....several options Look at emerging technologiesStay open for changes and surprisesCost efficiency is key here !! Final decisionwill be rather late PCIe-AS Whatever the implementation is, it will be called Ethernet... Ethernet CBM FEE/DAQ/Trigger - System and Network Architecture, Walter F.J. Müller, GSI

More Related