DiFX: Software Correlation at Swinburne for the LBA

1. DiFX: Software Correlation at Swinburne for the LBA Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

2. Outline • History • Correlator architecture • Capabilities • Computing environment and resources • Usage inside and outside LBA • Future improvements

3. Correlator History • Commenced with my PhD thesis (2005) • Preliminary work (XF) undertaken in 2004 by Craig West • First science late 2005 • Conversion to distributed system 2006 • Verification with LBA and VLBA correlators, code released late 2006 http://astronomy.swin.edu.au/~adeller/software/difx/

4. Correlator Architecture Written in C++, MPI for message passing, Intel IPP for vector arithmetic

5. Capabilities • Supports LBA, Mk5* (K5->LBA converter) • Pulsar gating, binning or matched filter • Arbitrary time/frequency resolution • Presently, visibilities in RPFITS format • eVLBI: data can be streamed from network socket directly into correlator • Also used for real-time fringe checks *export to linux filesystem

6. The Swinburne supercomputer • Swinburne supercomputer presently consists of ~300 P4 processors • Typically “reserve” ~30 processors for VLBI after observing run • Next month: complete replacement of cluster with ~140 dual CPU, quad core nodes, >1100 cores total

7. Computing requirements • DiFX requires little RAM (except extreme cases): processor speed, cache and SIMD support essential • Performance scales linearly with aggregate bandwidth, near-linear with number of stations • Realtime example: Max sensitivity LBA (6 x 1 Gbps) requires ~25 new nodes

8. Typical LBA usage • Required to correlate ~5 days of 256 Mbps before next session (2 months) • For each experiment, automatically select and correlate fringe-finders, solve clocks, followed by full correlation. • Using ~30 (old) machines, high data rate experiments take 2x observe time

9. Non-LBA applications • New geodetic array in Australia/NZ to use software correlator: modifications to control, output format • VLBA is trialling software correlator to run in parallel with hardware correlator: experiments with specific requirements, testing high data rate

10. Future improvements • Current version is production: future improvements will be incremental • Direct read from MkV • Alternative output formats: AIPS++ MS, FITS-IDI, UVFITS • Support in AIPS: tasks like CVEL currently confuse with LBA S2 (XF)

12. RA Dec Interferometry & Correlators Interferometry: delay signals from two dishes to common reference and multiply Common signal, independent noise: average improves S/N Earth rotates, interferometer samples visibility: Fourier transform of sky brightness

13. u () v () Interferometry & Correlators Finite bandwidth, not monochromatic, therefore visibility varies across band To get v(), XF correlator accumulates lags, then FFTs FX correlator FFTs segments of baseband data, and cross-multiply/accumulates

14. Hardware vs software • Software correlator: program running on a computer cluster/supercomputer • Hardware correlator: ASIC boards, specialised data transport • Software is unclocked, could be faster or slower than real-time • No channel/integration time restrictions • Floating pt vs int calculations

15. Why software? • Flexibility - you can do things that are impossible with a hardware correlator • Rapid (and cheap) development • Add-ons MUCH easier in software • Compatibility • Expandability • For me: Allow disk-based correlation, and improved pulsar binning (sensitivity)

16. Baseband data DataStream 1 Core 1 DataStream 2 Core 2 … … DataStream N Core M Visibilities Timerange, destination Source data Master Node DiFX (Distributed FX) MPI is used for inter-process communications

17. DiFX (Distributed FX) • Configured by text files (like jobscripts) • Delay modelling - CALC 9 (separate) • Output: RPFITS (built on-the-fly) • Arbitrary time/frequency resolution • Arbitrary pulsar binning (incoherent dedispersion) - allows weighted bin sum • Real time LBA @ 1 Gbps: 100-200 CPUs

18. DiFX (Distributed FX) • Verification: recent successful comparisons with LBA and VLBA

19. LBA Science • All require one or more of flexibility, high time/frequency resolution, or sensitivity • Wide field VLBI (Lenc & Tingay) • Masers (Horiuchi) • CDF radio counterparts (Norris et al.) • RRATs (Kramer et al.) • eVLBI (Phillips et al.) • Pulsar parallax (me)

20.  t Worldwide science • So far, motivated by very high frequency resolution, or flexibility and minimum effort for new system • Pulsar scintillation (Brisken) requires extreme frequency resolution (244 Hz channels over 32 MHz bandwidth) • Geodesy (MPIfR, Bonn, Germany) • Geodesy, new Australian array (NCRIS)

21. Status: Correlator code • “Correlation” code completed and verified - now finishing GUI and packaging for public release • VLBA, MPIfR and others keen to continue trialling code once released • Should be online within a month • PASP paper submitted simultaneously

22. Status: Observing and reduction • Four sessions of eight allocated, two observed - applications for 2007/2008 year (four more sessions) soon • Both observed sessions correlated and verified • Starting to work on atmospheric/ ionospheric compensation and pipeline, crucial for efficient, accurate astrometry

23. Roadmap to submission

24. Conclusions • DiFX is a general purpose software correlator, publicly available soon • Already used with success in Australia by the wider VLBI community, and generating interest internationally • Pulsar parallax program is underway, with the bulk of observations to come • Transitioning from code to science

26. Per station  N Per station  N2 Interferometry & Correlators • Major operations performed by FX software correlator: • Delay • Unpack quantized data to float • Fringe rotate • FFT • Correct fractional sample error • Cross multiply and accumulate A B Datastream