Recent results from the Swinburne supercomputer VLBI software correlator(s)

1. Recent results from the Swinburne supercomputer VLBI software correlator(s) Steven Tingay, Craig West, Adam Deller, Shinji Horiuchi Centre for Astrophysics and Supercomputing Swinburne University of Technology 4th eVLBI Workshop: July 12 – 14 2005 ATNF Marsfield http://astronomy.swin.edu.au

2. Outline of talk • Goals of project. • Why correlate in software? • Details of the observing/correlator setup. • Performance of the software correlator. • Availability of system to proposers. • Results for Vela pulsar. • Results for J0006-0623. • Results for PKS 0438-436 and PKS 0405-385. • OH masers • Wide-field imaging, other pulsars, “Double pulsar” observation on Saturday (ATCA/Parkes) • Summary.

3. Goals • Use supercomputers to demonstrate SKA-related techniques and technologies: process in software the sampled, digitised output of single dish and interferometric radio telescopes(VLBI/eVLBI!!); • Develop these techniques and technologies and integrate them into existing radio astronomy National Facilities; • Activities funded by State and Federal bodies and industry partners, in close collaboration with the ATNF, University of Tasmania, the Auckland University of Technology (New Zealand), and the University of Western Australia.

4. Apple Xraid disks Multibeam correlator Board (application- specific device) Dell server-class PC (generic processing device) RF electronics Sample/ Digitise 101001010… Δν = 16 MHz (56 – 72 MHz) ATCA VLBI system ~ few GHz Beowulf cluster RF electronics Sample/ Digitise 101001010… Δν = 16 MHz (56 – 72 MHz) Parkes Cray XD-1

5. Why correlate in software? • Moore’s law over the last ten years has made it possible – hopefully Moore holds on a bit longer; • Short development time for code – well known problem, well defined algorithms; • “Embarrassingly parallel” computing problem – suits parallel, multi-processor machines very well – strong cross-fertilisation with HPC and ICT sectors (LOFAR/Blue Gene). Vendor interest in the problem; • The algorithms are very flexible and it is easier to alter the code to reach new regions of parameter space than alter correlators with fundamental hardware constraints; • Software correlators suit small to medium sized arrays.

6. Details of the VLBI setup • PC-based data recorders (see Chris’s talk) • Dell 1600 SC or similar (now superceded); • Metsahovi Radio Observatory “PC-EVN” VSI boards; • Apple Xraid mass storage (5.5 TB with latest disks) • Swinburne Beowulf cluster supercomputer • 304 P4 processors; • Gigabit ethernet; • 6 x Apple Xraid units; • Cray XD-1 chassis • 12 Operton processors; • 6 Xilinx FPGAs (ultimately PGPUs as well) • Co-owned with University of Western Australia and ATNF

7. Details of the software correlator • Currently runs on Beowulf cluster under MPI; • Uses Intel Performance Primitive (IPP) libraries for optimised cross-correlation and Fourier transform routines; • First two versions of code implemented as XF correlators – next generation will be FX (see Adam’s talk next); • Complex correlation (multithreaded on single processor PCs); • Post-correlation correction of fractional sample errors on timescales of ~few ms; • Apply system temperature and gain information on the fly, standard FITS output (AIPS, AIPS++, MIRIAD, DIFMAP etc); • Arbitrary correlator integration time (down to microseconds) and number of frequency channels (correlated up to 16000 for narrow band maser observations) – applications for pulsars, wide-field imaging, and super-high spectral resolution masers for example; • Correlator is currently much slower than real-time, depending on the required output (getting much faster – see Adam’s talk); • Trade off unique capabilities with speed performance!

8. Speed performance of current XF correlator * system is i/o bound. Recent To correlate full Australian LBA: improvement to i/o handling gives ~x2 speed increase. 6 antennas = 15 baselines; 8 x 16 MHz (4 x 16 MHz at RCP; 4 x 16 MHz at LCP); 128 channels; 25 ms integration time; 12 hour observation; All polarisation products; 300 processor cluster; ~180 hours For next generation FX correlator, see Adam’s talk, next. Faster! Stay tuned for XD-1

9. Availability of the system • We can now correlate in software VLBI observations at up to x4 the bandwidth of the S2 tape-based VLBI system (factor of 2 improved sensitivity): • Aim for 1 Gbps operation (x8 bandwidth) in real time via optical fibre in 2006 (SKA demonstration) • Data are correlated in software on Swinburne supercomputer – eventually on the end of AARNet; • New facilities are available as part of the VLBI National Facility from June 2005, supported by the MNRF team at Swinburne, the ATNF, and other VLBI partners (first proposals received): See http://www.atnf.csiro.au/observers/apply/avail.html • Attempting first Trans-Tasman VLBI with Auckland University of Technology facilities; • Fringe-check software (uses software correlator) improves VLBI reliability – see Chris’s talk

10. Vela pulsar 2.3 GHz observations, Parkes to Tidbinbilla (test the models and analysis of Gwinn et al. 1997, 2000) Average pulse profile. 10 bins over on-pulse region. Bins of ~1 ms in width. Can generate arbitrary number of bins and bin width. Coherent de-dispersion prior to correlation. Analyse the amplitude statistics in each bin and fit scintillation models to estimate size of pulsar emission region. Work in progress.

11. Results for J0006-0623, PKS 0438-436, and PKS 0405-385 2.3 GHz (4 x 16 MHz) ATCA, Parkes, Hobart, Ceduna, Mopra (LBAHDR) Hartebeestoek (Mark5) Kashima (K5, 256 MHz - digitally filtered in software to match 16 MHz band – Hiroshi Takeuchi, July 2004, NICT News) 2 milliarcsecond resolution Science paper in preparation

13. Quasar PKS 0438-436: 2.3 GHz (4 x 16 MHz) ATCA, Parkes, Hobart, Ceduna, Mopra, Tidbinbilla (LBAHDR) 10 mas resolution Quasar PKS 0405-385: Famous intraday variable source (Kedziora-Chudzer et al. 1997) 2.3 GHz (4 x 16 MHz) ATCA, Parkes, Hobart, Ceduna, Mopra, Tidbinbilla (LBAHDR) 10 mas resolution

14. OH masers • 4 MHz • 16384 frequency • channels = 244 Hz/ch • 1 second of data • LL polarisation • Parkes – ATCA • 1720 MHz

15. Widefield imaging/other pulsar experiments • Observation of NGC4945 in May to show that software correlator can be used for wide-field VLBI (2.3 GHz; 4 x 16 MHz). • Observation of millisecond pulsar scintillation (GBT, Effelsberg, Arecibo; MarkV – correlated in software at Swinburne (project proposed – pending approval). • Observation of the “double pulsar” on Saturday (this Saturday!: Parkes/ATCA/Hobart; 1.4 GHz; 8 x 16 MHz – Huygens mode) • Pulsar “eclipses”; • Parallax/proper motion; • Pulsar nebula. • Phase referenced

16. Software correlator for realtime fringe checking (see Chris’s talk)

17. Summary • Swinburne software correlator is operational and available as part of the Australian VLBI National Facility; • Capable for multiple recording formats (LBAHDR; Mark V; K5 ……. • The next generation FX version will be much faster than our current XF – real-time Gb?; • Software correlators are: • Relatively easy to develop – good for niche users; • Capable and flexible – good for niche science; • Relatively slow – compared to dedicated hardware; • Great fun!!