The IceCube DAQ

1. The IceCube DAQ Kael Hanson Université Libre de Bruxelles IceCube Mid-Term M&O Review Madison, WI 15 May 2013

2. Outline • Summary and conclusions • DAQ technical description • SNDAQ – the online, near-realtime SN trigger • Team organizational structure • Metrics • Accomplishments • Future HANSON - IceCube DAQ : IceCube M&O Mid-term Review

3. Conclusions & Message • DAQ developers continue to maintain high-quality software and hardware product which is at the base of the science coming out of IceCube. • However, manpower, in particular hardware/firmware, is sub-critical for large-scale enhancements or extensions to the system. • Moreover a major risk is the loss of a core developer which would have an impact on even maintenance. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

4. Under the hood DAQ Technical HANSON - IceCube DAQ : IceCube M&O Mid-term Review

5. What does it do the DAQ? • In-DOM trigger / digitization (VHDL firmware) • Data communication over 0.5 – 3.5 km twisted pair @ 1 Mbit/s (firmware / software layers) • Online timing calibration (RAPCal: firmware at protocol level; software processes TCAL packets) • Data stream aggregation (DOM-to-string then string-to-array, Java) • Array-wide triggering in software (Java) • Event assembly and handoff to PnF (Java) • Network-distributed control framework (Java/Python) HANSON - IceCube DAQ : IceCube M&O Mid-term Review

6. DOM Mainboard • Core of the data acquisition • Two 300 MSPS ASICs • 40 MHz pipelined ADC • PMT pulses acquired time-stamped with local clock • Altera Excalibur FPGA with hard core ARM CPU • 4 W power dissipation HANSON - IceCube DAQ : IceCube M&O Mid-term Review

7. COMMs and RAPCal • DOM Communications • Custom designed baseband signaling @ 1 Mbit/sec • Two channels / twisted pair • Surface endpoint is DOR card – PCI card which supports 8 channels • 8 DOR cards per DOMHub  1 string per hub • Time Synchronization • 1 Hz RAPCal sequence: exchange pulses – stamp departure / arrival on each end • Clock relative frequencies calibrated by taking difference of T0’-T0 and T1’-T1 • Clock offsets from common mid-point time (assumes symmetry in circuit) • Essentially analog system – subject to noise on line (EMI susceptibility) • Precision of ~ 1 ns achieved nonetheless. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

8. DAQ Schematic 1 MB/s per string 10 MB/sec N.B.: Control framework omitted for clarity HANSON - IceCube DAQ : IceCube M&O Mid-term Review

9. StringHub Internals HANSON - IceCube DAQ : IceCube M&O Mid-term Review

10. DataCollector HANSON - IceCube DAQ : IceCube M&O Mid-term Review

11. Merge and Sort HANSON - IceCube DAQ : IceCube M&O Mid-term Review

12. Trigger HANSON - IceCube DAQ : IceCube M&O Mid-term Review

13. Current Trigger Manifest HANSON - IceCube DAQ : IceCube M&O Mid-term Review

14. Event Builder HANSON - IceCube DAQ : IceCube M&O Mid-term Review

15. Really, an online trigger … Supernova DAQ HANSON - IceCube DAQ : IceCube M&O Mid-term Review

16. Detecting SN Neutrinos with IceCube • In the Ice • GC type SNe expected / observed (SN1987A) to emit intense burst of low-energy neutrinos. • IceCube cannot reconstruct neutrino interactions in ice on event-by-event basis, however very large target volume yields millions of interactions when shockwave crosses – O(10) s. • Monitor background counting – GC SNe produces 106 excess counts. • Discriminator crossings counted by FPGA scalers in 1.6384 ms • 250 µs artificial deadtime to reduce correlated optical noise • At the Surface • Scalers readout 1 Hz as separate data stream from waveform hits – no Q, T. • Aggregrated streams written to disk files handed off to SNDAQ, a realtime trigger • Scaler stream analyzed within 10 min of acq, high significance events trigger alarm the highest of which are sent to SNEWS HANSON - IceCube DAQ : IceCube M&O Mid-term Review

17. The Role of Muons in SN Alarms IceCube Internal Memo 201107001 • Nature of noise in IceCube notPoissonian even after suppressing the correlated DOM noise: CR µ activity fluctuations broaden SN significance distribution. • This effect can be subtracted somewhat by analysis of the muon DST data HANSON - IceCube DAQ : IceCube M&O Mid-term Review

18. Integration with IceCube DAQ • SNDAQ is a separate system from DAQ • Different developers (Mainz) • Different language – C++ / ROOT • Until recently there has been only the scalers files interface with the DAQ and there has been no means to readout IceCube in response to supernova triggers. • This last point is discussed further in the section on HitSpooling. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

19. The DAQ Team HANSON - IceCube DAQ : IceCube M&O Mid-term Review

20. DAQ Effort in M&O HANSON - IceCube DAQ : IceCube M&O Mid-term Review

21. DAQ H/W + F/W + S/W • DOMs / Hubs: J. Kelley (WIPAC) • DOM firmware: T. Stetzlberger (LBNL) • DOR firmware: K.-H. Sulanke(DESY-Z) • DOR driver: John Jacobsen (NPX Designs) • DAQ Software • Dave Glowacki (UW Madison) • Matt Newcomb (UW Madison) • David Heereman (ULB) • Kael Hanson (ULB) • Others have contributed significantly to triggers • Interface to I3Live: (M. Frère) HANSON - IceCube DAQ : IceCube M&O Mid-term Review

22. DAQ Quality Control • Weekly CCB phone call since 2005 to discuss development status / plans • Software releases under strict management of Dave Glowacki. • Issue tracking • Automatic linting tools used • Regression and integration testing on SPTS northern system before deployment at Pole • 24-hour test runs made at Pole HANSON - IceCube DAQ : IceCube M&O Mid-term Review

23. Metrics HANSON - IceCube DAQ : IceCube M&O Mid-term Review

24. DAQ Lines of Code HANSON - IceCube DAQ : IceCube M&O Mid-term Review

25. Mantis Issues Metric HANSON - IceCube DAQ : IceCube M&O Mid-term Review

26. DAQ Uptime HANSON - IceCube DAQ : IceCube M&O Mid-term Review

27. Major Bugs • Hardware: • DOM pulse droop • Requires a lot of calibration • Firmware: • Deadtime bug • LBM overflow “bug” • Software: • Subthreshold triggers • Triggers can be fragile • Flasher instabilities caused some delay in calibration • Dropped DOMs (software/firmware?) • None of these bugs have prevented science but … • Do result in additional work for data processors/analysis • Why don’t we have many bugs? It’s testing, testing, testing … • DOM engineering • Good software development rigor • Excellent team • And a little luck too, OK. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

28. Recent Accomplishments HANSON - IceCube DAQ : IceCube M&O Mid-term Review

29. Short List of Past Accomplishments • Riding on top of delivery of new triggers almost every year, there is also … • Support for readout of all hits (isolated and LC-triggered) IC59+ • Improvements in data format packing (was choking I/O in DAQ-to-PnF handoff) • Better, more manageable configuration system • Programmed digitizer pedestals now remove any ambiguity in determination of baselines. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

30. DAQ Low Level • Recent DOM firmware update (J Kelley) to fix pedestal wobble and also improved code to reject light contamination during pedestal measurement on run initialization. • Improvements (still to be rolled out) in DOM/Hub messaging – reduce polling overhead. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

31. HitSpooling • The DOMHub reads out the DOMs, parsing the records into hits. It then forms trigger summaries for LC hits and buffers the full hit until EventBuilder asks for it or says it’s OK to discard. • The buffering is done in memory – approx 1 min buffer time is available. • For systems such as SNDAQ it would be nice to be able to readout full IceCube during a large (~minutes) window around event. • Disk resources on hubs are much more plentiful: depending on generation of hub there is between 50 GB and 2 TB free space. Buffering to disk would allow days of depth of the complete record of IceCube PMT pulses! • This could be useful for alerts such as GCN which can have long latency. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

32. HitSpooling – DAQ Side HANSON - IceCube DAQ : IceCube M&O Mid-term Review

33. Client Side (D. Heereman) • The DAQ has been writing hit spool files since January 2013. There does not appear to be any issue with stability of DAQ or increased disk wear that we have observed thus far. • The other side of the system is a facility which can automatically grab data from these files and deliver to a client wishing access. • Operation • Any client (SNDAQ is only one currently) sends time interval of data to extract and network copy destination to interface running on experiment control computer. • This interface forwards message to workers running on each DOMHub. These workers identify the correct file(s) and copy the file to destination. • 15 s granularity because files are copied atomically. • Entire process of trigger, copy, satellite transfer in place since April 2013. Currently extracting 90 s of data around SN alerts > 8 (about 0.5 per week). HANSON - IceCube DAQ : IceCube M&O Mid-term Review

34. HitSpooling Data Interface HANSON - IceCube DAQ : IceCube M&O Mid-term Review

35. Uses for HitSpooled Data • Improved security against information loss during DAQ crashes. • Full record of IceCube PMTs during SN shockwave: • Get around scaler saturation for nearby SNe • In principle very fine time resolution of structure in neutrino light curve • Subtraction of correlated noise, subthreshold muons • <E> determination for SN neturinos. • Using HS data as test input for playback on SPTS HANSON - IceCube DAQ : IceCube M&O Mid-term Review

36. SWITCHRUN Technology • Many systems became dependent on the 8 hr run structure to organize run metadata. • Most of the time IceCube DAQ is just cycling the run without changing the configuration. • This cycling takes 1 min – 3x per day is 0.2% deadtime – a significant fraction of our 0.8% downtime! • The last DAQ release includes support for run number switching without actually interrupting the DAQ stream. • A number of changes in DAQ and out to support bookkeeping of run start/stop times. We are being very cautious with the roll out which should happen imminently. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

37. Improved Triggers • Trigger system underwent a substantial rewrite to cleanup code and improve the performance of the triggers. • This includes finally threading the dozen algorithms we currently have to take advantage of the multicore architecture on which the triggers run. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

38. DOMHub Upgrade Near-Term • DOMHub computing platform not upgraded since initial delivery in 2004. • SBC was chosen for low-power not performance. • Subsequently, DOMHub assigned compute tasks well in excess of original plan. • While we have managed to keep the system stable, all stops are pulled out - not much room for further optimization. • 2012 – JK/JJ/KH identified SBC which nominally worked as drop-in replacement • At same time, larger disk, high-efficiency ATX power supply upgrades • 10 hubs upgraded at Pole as test 2012-13 • CPU usage now at 40% vs 99% • System load dropped from 60 to 2.2! • Next year will upgrade remaining hubs • Looking for easier ATX PS upgrade path. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

39. DAQ Future HANSON - IceCube DAQ : IceCube M&O Mid-term Review

40. Future DOMHub Upgrades • The upgrade next year will include only SBC and disk drives with a few test deployments for higher efficiency ATX power supplies. • We estimate being able to save ~ 2 kW by moving to 80% efficient supplies – about $30k per an in fuel. • In the far future (4-5 years) we have considered a more radical DOMHub upgrade which would include redesign of the comms system • Higher channel density / smaller ICL footprint • Dramatic power savings (10 kW) • Better maintainability – our biggest source of downtime is hub hardware problems. • However, SBC upgrade was wildly successful. We may benefit to wait and see what direction(s) IceCube will turn to now that the science is getting really interesting. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

41. Other Future Projects • HitSpooling was the first step in the direction of separation of DOM readout and hit buffering from higher level things like triggers and event assembly. Eventually we would like to see a loosely coupled system here. • Triggers which see all hits and not just LC hits. This was at one time very difficult computationally. With the improved, threaded trigger system we should revisit – perhaps minor performance tweaks to DAQ code can get the job done. • More monitoring of DAQ by DAQ. HANSON - IceCube DAQ : IceCube M&O Mid-term Review

42. The Real Summary • DAQ/SNDAQ working, delivers high-uptime and is meeting science needs of experiment. • Improvements in DAQ/SNDAQ continue to be active even as software is in mature state. • DAQ team very lean and efficient but there are substantial risks dealing with this small size. HANSON - IceCube DAQ : IceCube M&O Mid-term Review