Data Acquisition Systems for Future Calorimetry at the International Linear Collider

1. Data Acquisition Systems forFuture Calorimetry at theInternational Linear Collider Matt Warren, on behalf of CALICE-UK Collaboration

2. Introduction • Building a DAQ for multiple ILC CAL sub-detector prototypes • Paying attention real ILC environment • EUDET Testbeam in 2009! (EU funded DETector project shares much of CALICE) • For economies of scale, we are attempting a generic DAQ for many (even SLHC??) • Modular/Generic Structure: • Generic readout system as much as possible • Detector specific interfaces only at ends of chain • Other ‘bespoke’ functionality in firmware • Commercial components and protocols where possible • Readout links use standard connectors and protocols • Based on PCs with PCIe cards (“backplaneless”) • Clock and Control attempts commercial hardware too • Extract clock and ‘fast’ signals from commercial signalling • Software generic for all detectors • Try use something off-the-shelf … BUT first, an introduction to the ILC CAL … Matt Warren - DAQ for Calorimetry at ILC

3. Time structure of bunches Trains of bunches Individual bunches ILC Calorimetry • ILC Calorimetry is dense and high granularity • squeezed between large tracker & expensive coil • >100M channels • No room for electronics or cooling. • Bunch structure interesting: • ~200ms gaps between bunch-trains • Trains 1ms long, 300ns bunch spacing • Triggerless – sample data from every bunch-crossing SO (the problem): • 100M channels, analog signals = front-end electronics inside detector • Results in high power density • but no room for cooling • Long gap allows chips a 1% duty cycle Solution: Power Pulsing HCAL ECAL M. Anduze Matt Warren - DAQ for Calorimetry at ILC

4. Sub-Detector Geometries (+ASICs) ASICS • Must share readout resource (daisy chain) • Bunch rate too high for instantaneous data transfer. • Too much chip resource to store all events SO: • ‘Auto-trigger’ – store only data over-threshold with pad id + (bunch-number) • <5kByte / bunch-train/ASIC HCAL half-octant ECAL Module-0 (reduced-Z octant) M. Anduze L = 150 cm ASIC (>100 in total!) Detector Unit (e.g. ECAL Slab) Matt Warren - DAQ for Calorimetry at ILC

5. Host PC Host PC ODR ODR PCIe PCIe DIF DIF DIF DIF Detector Unit Detector Unit Detector Unit Detector Unit DAQ architecture Detector Unit:Sensors & ASICs DIF: Detector InterFace -connects generic DAQ and services LDA: Link/Data Aggregator – fanout/in DIFs & drive link to ODR ODR: Off Detector Receiver – PC interface for system. C&C: Clock & Control: Fanout to ODRs (or LDAs) 50-150 Mbps HDMI cabling 1-3Gb Fibre LDA C&C Counting Room Detector Storage LDA 10-100m 0.1-1m Matt Warren - DAQ for Calorimetry at ILC

6. DIF (Detector InterFace)  DAQ • FPGA + detector hardware connected to Detector Unit. • Two halves – Generic DAQ and Specific Detector • 3 detectors: ECAL, AHCAL, DHCAL • 1 DAQ Interface! Focusing on the DAQ side: • From LDA, receive, decode/regenerate and distribute clocks, fast commands, config data and slow controls. • From ASICs, receive, buffer, package and forward data to LDA • ASICs power-up and read-out power-down in turn • ALSO: USB interface • Hardware designers already have one • DAQ plans to integrate for stand-alone testing DIF Detector Unit USB M. Goodrick e.g. ECAL Matt Warren - DAQ for Calorimetry at ILC

7. DIF DIF DIF DIF Detector Unit Detector Unit Detector Unit Detector Unit DIF-LDA link • Serial links running at multiple of machine clock • ~50Mbps (raw) bandwidth minimum • robust encoding (8B/10B) • HDMI cables/connectors interface. • Commercially available cables • Rated >300Mb • Even halogen free available • Signals (ideally just TX/RX but …): Clock (diff) Control/Fast (diff) Very Fast (diff) Data (diff) single ended aux x2 (or UTP) • LDAs serve even/odd DIFs for redundancy LDA LDA Matt Warren - DAQ for Calorimetry at ILC

8. LDA (Link/Data Aggregator) • Located as close as possible to DIFs • Shortest cables, but convenient location (space, cooling) • Supports as many DIFs as possible considering bandwidth and physical constrains • Ideally 50 (20Mbps/DIF) • Prototype will have 10 • Aggregates front-end data and sends it off-detector • Fibre optic link. 1-3Gbps, with SFP (see next slide) • Fanout C+C+C to DIFs • USB interface for stand-alone/top-of-chain testing LDA LDA • Currently using a commercial FPGA dev-board: Enterpoint Broaddown2 – Xilinx Spartan3-200 • With add-on boards for our needs • SPF+SerDes for ODR link • 10 HDMI connectors with clock fanout Matt Warren - DAQ for Calorimetry at ILC

9. Host PC ODR PCIe ODR (Off Detector Receiver) + Link • Receives module data from LDA • PCI-Express card, hosted in PC. • 1-4 links/card (or more), 1-2 cards/PC • Buffers and transfers to store as fast as possible • Fibre optic link to detector via SFP modules (std networking hw) • Currently GigE (1.25Gb), but could higher and use different proto. • Sends controls and config to LDA for distribution to DIFs • Interfaces to C+C for synchro running • Goal to send clock and prompt controls over optic link too • Reset and reprog FPGAs • Performance studies & optimisation on-going (see next slide): • Bottleneck in writing data to disk. C&C Storage Expansion (e.g. 3xSFP) SFPs for optic link Hardware: • Using commercial FPGA dev-board: • PLDA XPressFX100 • Xilinx Virtex 4, 8xPCIe, 2x SFP (3 more with expansion board) • Our own firmware and Linux driver software Matt Warren - DAQ for Calorimetry at ILC

10. ODR Throughput Measurements • All measurements: single requester thread, single IO thread (disk write), • Each event fragment written to a separate file. Data written to the localdisk (fs: ext3) • e.g. WORST CASE! ` Ethernet frame size Problem with test! A. Misiejuk NDG – Network Data Generator IDG – Internal (ODR) Data Generator NDG plot – between two separate machines, Gigabit, copper Eth on both sides Matt Warren - DAQ for Calorimetry at ILC

11. Host PC Host PC ODR ODR PCIe PCIe Clock & Control • C&C unit provides machine clock and fast signals to 8x ODR/LDA. • Logic control (FPGA, connected via USB) • Command encoders • Remote signal enable, clock selection • But capable of stand-alone, dumb mode • Provision for async scintillator type signals (VFast) • LDA provides next stage fanout to DIFs • Eg C&C unit -> 8 LDAs -> 10 DIFs = 80 DUs. • Signalling over same HDMI type cabling • Facility to generate optical link clock (~125-250MHz from ~50MHz machine clock) LDA Machine C&C Run-Control LDA • Commercial systems are not ideal here. • Looking at custom protocol on fibre optic link • Prompt signals and low jitter clock recovery needs further investigation Matt Warren - DAQ for Calorimetry at ILC

12. Software and Operation Software: Looking for OTS software to cover both slow and fast controls: Early days – examining EPICs, ACE and DOOCS (the current favourite) - DOOCS is open source, actively developed, slow and fast controls, and already used by ILC community Operation: Two modes: 1) Configure: PCs controlled over network to send configuration to LDAs and DIFs 2) Run: • LDA/DIF set to data-taking mode • ODR configured for data reception, control handed to central. OR ODR needs no control – simply waits • Bunch-train starts/stop signals sent to LDAs control data flow. Fairly autonomous system (i.e. no trigger!) Matt Warren - DAQ for Calorimetry at ILC

13. Summary (an example – AHCAL) Off-Detector DAQ LDA DIF Detector Unit P. Göttlicher, DESY Matt Warren - DAQ for Calorimetry at ILC