xNTD/SKAMP/LFD Correlator

1. xNTD/SKAMP/LFD Correlator 4th RadioNet Engineering Forum Workshop Next Generation Correlators for Radio Astronomy and Geodesy 27-29 June 2006, Groningen, The Netherlands, John Bunton CSIRO ICT Centre, Sydney

2. Common Design • Developing a set of technologies that will be used in xNTD, SKAMP3 and LFD • Common correlator, VHDL code and hardware elements • This talk mainly the correlator and xNTD beamformer • Team (SKAMP3, xNTD and LFD) • Uni Sydney/ ATNF - Ludi de Souza, • Uni Sydney Duncan Campbell-Wilson, John Russel, Chris Weimann, Adrian Blake • ATNF/ ICT Centre John Bunton, Jaysri Joseph • MIT Roger Cappallo, Brian Fanous • ANU Frank Briggs • and growing

3. xNTD specification • Connected element telescope with 20-30 antennas within ~10km area • Each antenna generates 32 dual pol. beams giving a field of view of ~30 square degrees – frequency independent • Dynamic range ~50dB (8-bit A/D) aided by low RFI environment at Mileura WA, >60dB filterbanks • Frequency range below 1.7 GHz with 300 MHz BW

4. xNTD Correlator • FX correlator, FPGA based using polyphase filterbank • Large design with 20-40 antennas x ~32 beams • Input to beamformers 1.5 THz, to correlator 500 GHz (16bits/Hz) • 6,000-15,000 baselines, each full Stokes • ~1000 Tops/sec (mainly beamformer/filterbanks) • Output data rate 1-2 Gcorrelation/visability set • Full 300 MHz processed to full frequency resolution • ~50,000 channel – 6 kHz resolution (no options) • BUT • Low resolution and continuum by averaging channels • Daisy chain data for special processing • Later can trade bandwidth for higher resolution or time sampling

5. Design • Beamformers and filterbank • ~200 inputs generating 32 dual pol. beams • Signal transport from feeds optical analogue (TBD) • Beamformer location TBD • Design similar to SKAMP3 filterbank (Ludi’s talk) • Correlator based on correlation cell • Highly configurable as to number inputs • If the beamformer is at the antenna crossconnect in fibre network and O/E interface • Needs data re-ordering (Ludi’s talk)

6. xNTD Beamformer • Beamformer input • 96 Dual polarisation inputs • 300 MHz bandwidth • About 1 Terabit/sec input data rate per antenna • Each of 32 output beams is the sum of 20 to 96 inputs depending on frequency • Would like to have data from all inputs to each unit that generates a beam • Instead “Divide and Conquer” • Use the same topology as SKAMP3 filterbanks (Ludi’s talk)

7. Filterbank to Beamformer

8. Correlator • The challenge was to develop a correlator concept the could be used in xNTD, SKAMP3 and LFD - 20-30, 192, and 512 antennas • Systolic array too inflexible • xNTD 760 correlations/beam • LFD 500,000 correlations • Approach developed – Correlation Cell • Combination of multiply-accumulate and storage • Each cell handles 256 correlations at a time • 49,000 correlations per FPGA simultaneously • 512 time sample short integration on chip

9. Correlation Cell • Input 16+ pairs of data • 4bit complex multiply in 18-bit multiplier • Accumulation to block RAM • Calculate 256 correlation, 512 successive time samples • Data reordering in filterbank • xNTD 4-7 cells for all correlations • 30-70 MHz BW per FPGA • All baselines LFD 12, SKAMP3 1.5 FPGAs • 1.2-1.5 MHz of bandwidth

10. 22 antenna xNTD configuration • Store 22 or 22 pairs per correlation cells • Within group 231 correlation • Between groups 484 correlations • Need two cells • 48 sets of 22-antenna correlators per FPGA (Virtex4 SX35) • Clock cycles 242 for 44 inputs = 5.5 clocks per input • Input data rate 48/5.5 = 8.7 bytes/clock • As systolic array 192 multipliers gives 13x13 array • Input data rate 3 times higher at 26 bytes/clock

11. Board Manufacture Simplification (1) • Manufacture of correlator board a major task • Examples SKAMP1, EVLA … • Correlation cell reduces input data rate into correlation chip • For xNTD correlation cell data rate 3 times less compared to systolic array • For individual correlation cell 2 sets of 16 inputs requires 256 clock cycles to process. • Data rate reduction up to a factor of 16 • This value approached for SKAMP3 and LFD • For xNTD need ~5 byte wide busses into each correlator FPGA. (input data rate higher than correlator clock)

12. Board Manufacture Simplification (2) • Correlation cell also reduces data duplication • SKAMP1 4x4 systolic array, EVLA 8x8 systolic array • Data duplication 8 in EVLA, higher in SKAMP1 due to array reuse • Each Correlation Cell process 256 correlations at once • Can reduce size of systolic array by sqrt(256)=16 • No data duplication on board for up to 150 antennas • Data duplication none for xNTD, 1.5 for SKAMP3, and 3.5 for LFD, LFD 12 FPGAs data input 1/sqrt(12) of total • Correlation cell leads to a large simplification of correlator board

13. Putting it Together – The SKAMP protoype Correlator interface Correlations Routing Long Term Accumulations Autocorrelations

14. Input, Daisy Chain, Route, & Autocorrelate • Two FX20 • Interconnection for high antenna number designs • Input 16 rocket I/O on unidirectional Infinband • Output 16 Rocket I/O unidirectional Infiniband • Can daisy chain modules for reuse of data in further processing modules or conform to standards, Infiniband … Two FX20

15. Compute Engine • Six SX35 FPGAs • Input 24, Output 18 LVDS per FPGA • 1152 Correlation cells total • Up to 294k correlations on board at a time (256 per cell) • Data re-ordering in filterbank to achieve • Process of 512 time values for each frequency channel • Then dump to LTA

16. Long Term Accumulator • Number of Correlation require DRAM for storage • Data rate requires two DIMM modules for three SX35 Three SX35 Two LX15 + DDR2 DIMM Half of Correlation and LTA hardware

17. Estimated Performance • Correlator board clock rate 330MHz, 192 cells/FPGA, 6 FPGAs • Board processing rate 400GCMACs/s = 2.8Tops/s • Power consumption ~100W • Power efficiency 0.25W/GCMAC (4bit FX) • Filterbank board 3.2Gsample/s, two polyphase filterbanks ~32 operations per sample • Board processing rate 100Gops/s (18 bit) • Power consumption ~60W • Power efficiency 0.6W/Gop

18. Conclusion • Common hardware, hardware modules and VHDL for xNTD, SKAMP3 and LFD • SKAMP3 in the lead with filterbank and correlator hardware well on the way • Initial Manufacture this year • Correlator common to all using correlation cell to gain required flexibility • xNTD beamformer hardware starts July 06 • LFD receiver will piggy back on this • Developing international project - distributed design team