1 / 12

High Bandwidth Data Acquisition and Network Streaming in VLBI

High Bandwidth Data Acquisition and Network Streaming in VLBI. Jan Wagner, Guifré Molera et al. TKK / Metsähovi Radio Observatory. Introduction. Basic e-VLBI Data Acquisition. Basic: 8–16 stations, rates 128–512 Mbit/s; 24h 1– 4 TB/station, all to correlator

ocean
Download Presentation

High Bandwidth Data Acquisition and Network Streaming in VLBI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. High Bandwidth Data Acquisition and Network Streaming in VLBI Jan Wagner, Guifré Molera et al. TKK / Metsähovi Radio Observatory

  2. Introduction Basic e-VLBI Data Acquisition • Basic: 8–16 stations, rates 128–512 Mbit/s; 24h 1–4 TB/station, all to correlator • VLBI: observed galaxy’s noise recorded to disks, shipped to correlator • e-VLBI: transfer station data via fast research networks, two alternatives: • post-observation: record locally, transfer to correlator later • real-time: stream to the correlator for temp storage or processing • EXPReS: funds to connect stations at 1G/10G • 1G fiber up to capacity with Tsunami UDP (2006) or JIVE vlbi_udp (2007) real-time transport protocols • ToDo: new system for >1 Gbps rates! 1Apr08, JIVE vlbi_udp incoming aggregate rate record EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  3. Introduction Goals for High(er) Bandwidth DAQ • Observe a broader radio spectrum • broader spectrum => higher data rate • correlation result and final image improves! • high EXPReS goal 1/2/4/8 Gbps per station • Higher rates => need at least: • new analog-to-digital converter system (current 10+ years old) • faster, higher capacity local/remote data storage systems • suitable transport protocols - stations to correlator • Ultimately e-VLBI data processing in real-time • get rid of high capacity station data storage • JIVE: hardware correlator successful in many ≤512 Mbps real-time runs • distributed software correlators more natural with data streaming C. Phillips, e-VLBI Scientific Benefits, ATNF CSIRO memo, 2004 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  4. Development Environment High Bandwidth DAQ Development DAQ steps: Digitize wide-band signal – Stream over 10G – Capture/store with PC(s) EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  5. Basic Data Acquisition System DAQ – Digitizing • UC Berkeley’s iBOB FPGA hardware platform • Virtex2 Pro FPGA with PowerPC405 • two CX4 10G ports • 1..2 Berkeley iADC expansion boards Atmel AT84AD001 two channels, 8-bit, 1 Gsample/s => max 8+8 Gbps from one iADC => 32 Gbps ADC data, 20 Gbps net • Semi-ready component libraries from Berkeley for “faster” start of development • Analog signal digitizing setup: • iBOB FPGA reads raw 2-channel iADC data • FPGA processes data, implements network streaming => … EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  6. Basic Data Acquisition System DAQ – Streaming • TCP is not a good streaming choice • No unreliable best-effort delivery, no delivery time constraints • Real-time VLBI stream requirements: • data loss f≤10% is tolerable (impact is ), occasional re-order ok • data integrity non-critical: BER*2·10-9 frame CRC32, checksumless UDP • data must have time stamp: a 64-bit PSN sufficient, 35kYears@1Tbps,8K • congestion control: none, data thinning, change sampling mode • iBOB UDP streaming first implementation • iBOB has only 14MB RAM (≈30ms@4 Gbps)=> no retransmissions => plain UDP packetizer, 1 stream per channel • streaming configurable remotely via Telnet/ssh overall iBOB test design… => . EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  7. Basic Data Acquisition System DAQ – iBOB RF-to-10G Test Design EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  8. Basic Data Acquisition System DAQ – Storage / RAID Performance • Should use RAID-0 disk striping for best throughput • Hardware RAID e.g. PCIe cards, XRaid, VTrak, .. too expensive and slow • Software RAID baseline performance test on: ~1k€ workstation PC, Linux RAID-0 12 x SpinPoint F1, 9TB • Test method: • source buffer ~90% RAM size • sequential fwrite() to a single file • not using 10G / UDP yet • => Inexpensive 1–4 Gbps data storage system! EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  9. Basic Data Acquisition System DAQ – Stream Recording • Wrote simple UDP/raw capture programs udp2raid, raw2raid • Essential for performance in 10G stream recording: • 10G NIC throughput: • 4.7/9K jumbo frames, UDP checksums off • prefer larger UDP packets, larger OS buffers => Myri, Chelsio NIC PC iperf : 9.9 Gbps UDP, 0% loss • Application and disk throughput: • use multi-core system, try to utilize (nearly) all cores separate network and disk I/O into own threads => udp2raid, raw2raid currently min. 4 Gbps from 10G to RAID-0 => we have 1–4 Gbps iBOB stream to Abidal PC disk recording EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  10. Basic Data Acquisition System Fast “off-line” e-VLBI File Transfer • Transfer station files to where you process the data, one option: Tsunami UDP v1.1 fast ‘FTP’ and real-time (http://tsunami-udp.sf.net) • Weekly used in geodesic VLBI for ≤1 Gbps file transfer, occasional real-time • Current v1.1 version, first result on 10G LAN: • high CPU load, 3 – 3.5 Gbps to RAID, protocol rate limit 4 Gbps • Started v1.2 of protocol to use udp2raid improvements: • 4 – 4.4 Gbps to Abidal PC RAID • 7 Gbps to RAM (vs. UDT4-protocol 5.2 Gbps to RAM) => Improved rates for “off-line” e-VLBI EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  11. Summary Summary • Many European VLBI stations start to reach ~1 Gbps streaming with the old hardware • Basic iBOB design can further improve the rate: digitize and stream 1 – 20 Gbps UDP • A consumer ~1 k€ PC + 12 disks is capable of 4 Gbps storage:can capture the iBOB stream(s) to PC RAID • Tsunami v1.2 has improved file transfer rates • Can transfer from station to similar ≥4 Gbps PC elsewhere • ToDo: port real-time Tsunami server onto iBOB board EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

  12. Questions/Answers • Contact information Jan Wagner M.Sc./Researcher EXPReS (Metsähovi) jwagner@kurp.hut.fi • Additional Information http://expres-eu.org/ [note: only one “s”] http://www.metsahovi.fi/en/vlbi/ • EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #026642 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI

More Related