Technology and Functionality of the EVLA Correlator (Next Corr Workshop, June 27, 2006)

1. Technology and Functionality of the EVLA Correlator(Next Corr Workshop, June 27, 2006)

2. Outline • EVLA project overview. • Correlator capabilities. • Correlator architecture. • Technology. • Schedule. Next Gen Corr Workshop: EVLA Correlator—Carlson

3. EVLA Project Overview • NRAO: • Upgrade VLA from the current ~100 MHz BW to 16 GHz (8 GHz/pol’n). • New receivers: 1-50 GHz contiguous frequency coverage. • Replace waveguide system with digital fiber system. • 3-bit (8x2 GHz) and 8-bit (4x1 GHz) sampling. • New correlator (NRC Canada). Next Gen Corr Workshop: EVLA Correlator—Carlson

4. Correlator Capabilities • 16 GHz/antenna (8 GHz/poln). 32-antenna system…expandable. • Can tradeoff BW for #ants and #beams; can proc multiple narrow (<1 GHz) bands; VLBI ready. • 3/8-bit ADC (at antenna), 4/7-bit re-quantization and correlation…more bits/less BW possible. • All digital +/-1/32 sample delay tracking; 0.26 sec delay range. • 3-level phase rotation in correlator chip: sub-sample delay, LO offset removal, earth-rotation phase, bias/birdie/aliasing de-correlation. Next Gen Corr Workshop: EVLA Correlator—Carlson

5. Correlator Capabilities • 16k channels/baseline wideband; up to 4 Mchannels/baseline narrowband with “recirculation”. • Up to 144 tunable multi-stage digital filters per antenna…’slot’ restrictions in 1st stage of filter. • Use filter logic for sub-band multi-beaming and/or narrowband delay. • ~60 dB spectral dynamic range. • “WIDAR” technique (hybrid). Next Gen Corr Workshop: EVLA Correlator—Carlson

6. Correlator Capabilities • High-speed pulsar phase binning; 2000 bins/product, min ~15 usec bin width. • High-speed dumping. 1 Gbps Ethernet/board standard. Up to 10 Gbps/board. 25 Mbytes/sec to archive. • All digital phased output…1 GHz…expandable to full BW. • Unlimited sub-arrays. • Multiple VSI I/O. • Configurable for virtually any radio telescope configuration…including auto-correlation. • Uses VLBI standard frequencies (256 MHz…). Can incorporate VLBA into “excess” correlator capacity. Next Gen Corr Workshop: EVLA Correlator—Carlson

7. Correlator Architecture • Station Board: • 6U Fiber receiver mezzanine card (NRAO). • Cross-bar switch FPGA. • Delay module mezzanine card (+/-0.5 samples delay). • Wideband autocorrelator. • Multi-stage digital filters (1 FPGA per sub-band). • Cross-bar switch and sub-and multi-beaming delay memory. • 4 stages…128 MHz…31.25 kHz BW out. • Power measurements for sub-band stitching. • Real-time RFI blanking per sub-band. • CPU-settable scaling factors/re-quantization for max sensitivity/dynamic range. • State counts, lag-0 power, phase-cal. • Narrowband sub-sample delay with 16-step FIR interpolation. • Cross-bar switch and mux to 1 Gbps for data transport to Baseline Board. Next Gen Corr Workshop: EVLA Correlator—Carlson

8. Next Gen Corr Workshop: EVLA Correlator—Carlson

9. Next Gen Corr Workshop: EVLA Correlator—Carlson

10. Next Gen Corr Workshop: EVLA Correlator—Carlson

11. Correlator Architecture • Fanout Board: • Dual 1-8 fanout of signals (4-wafer ‘stack’, 1 sub-band, all basebands, one antenna) for baseline rack signal distribution. • Baseline Board: • 8x8 matrix of corr chips fed by 8 ‘X’ and 8 ’Y’ FPGAs (recirculation, phase generation, 7-bit handling, cross-bar). • Corr chip: 2048 c-lags, in 16 128 c-lag “cells”. Full 4-bit multiply, no trunc, 10-500 sec int. Cells can be concatenated. • Dedicated LTA FPGA +256 Mbit RAM for each corr chip; high-speed phase binning and recirculation. • Output 1 Gbps Ethernet via FPGA. • Can set corr chip and board for autocorr mode: 64 independent 2048 channel, 128 MHz autocorrelations. • Can process up to 32 stations (1024 correlations), 128 MHz, 1 poln product, at 64 channels/baseline. Next Gen Corr Workshop: EVLA Correlator—Carlson

12. Next Gen Corr Workshop: EVLA Correlator—Carlson

13. Next Gen Corr Workshop: EVLA Correlator—Carlson

14. Next Gen Corr Workshop: EVLA Correlator—Carlson

15. Technology • Use Xilinx Virtex-IV, and Altera Stratix II/GX/Cyclone FPGAs. • Standard cell correlator chip, 4 million gate, 0.13 m. PD~3.7 W each @ 256 MHz, 1 V. Use ‘Accel’ point-of-load reg to minimize power, track chip speed with time. • BGA packages throughout, some gull-wing memories/drivers. Next Gen Corr Workshop: EVLA Correlator—Carlson

16. Technology • 8 different boards in total: • Station Board, Baseline Board: 12U x 400 mm, 28 layer, 0.0035” trace-’n-space, 0.125” thick. Phasing Board: ~6U x 160 mm(?). • Fanout Board: 6U x 100 mm, with re-sync Stratix II FPGA. • Common Backplane/midplane (3U x 25 mm); signal I/O, power, ID…used everywhere…no monolithic backplanes. HM 2.0 mm straight-thru. All press-fit. • Delay module (~5”x3.5”)mezzanine card: FPGA+DDR SDRAMs • PC/104+ COTS CPU + PCMC card. • RPMIB (very simple; diodes, optos) for rack/fan power monitor and control Next Gen Corr Workshop: EVLA Correlator—Carlson

17. Technology • 1 Gbps station-baseline data transport on high-density Meritec 2 mm hardmetric cable assemblies. Make correlator highly configurable for various correlator configurations (e.g. e-MERLIN). • Use Altera’s 1 Gbps mux/demux with dynamic phase alignment…built into Stratix chips. • About 500 rack-to-rack cables, 2500 intra-rack cables (<1 m) in EVLA correlator. All cables plug into 3U x 25mm Common Backplane. Next Gen Corr Workshop: EVLA Correlator—Carlson

18. Next Gen Corr Workshop: EVLA Correlator—Carlson

19. Next Gen Corr Workshop: EVLA Correlator—Carlson

20. Baseline Board (12Ux400 mm) PCB-back Next Gen Corr Workshop: EVLA Correlator—Carlson

21. Baseline Board PCB-front Next Gen Corr Workshop: EVLA Correlator—Carlson

22. Thermal vias “pocket” for corr chip heatsink Next Gen Corr Workshop: EVLA Correlator—Carlson

23. Next Gen Corr Workshop: EVLA Correlator—Carlson

24. Next Gen Corr Workshop: EVLA Correlator—Carlson

25. Technology • Use central -48 VDC COTS power supply, DC-DC supplies on each board. Provides battery backup (5 min req). • 24, 24” racks (2.5’ x 3’ x 7’), each one holding up to 16 large boards in two crates, and Fanout Boards in 6U crate. • All hot-swappable, including fans. • On-line detection of communications errors, temps, voltages…remote shutdown/power cycle of any board through central PXI chassis/CPU. Next Gen Corr Workshop: EVLA Correlator—Carlson

26. Thermal mock-up test rack Next Gen Corr Workshop: EVLA Correlator—Carlson

27. Technology • Software: • Embedded PC/104+ COTS CPU on each large board. Cheap, fast, easily available, easily replaceable, many vendors. • RT Linux…”Unix-style” device drivers. • System…board…chip-level GUIs for initial testing. Useful during operations for continued debugging, low-level access. • MCCC—central host computer maps high-level requests from EVLA M&C to embedded processors, via XML messages. • CPCC—central power control computer. • Backend—array of COTS PCs connected to Baseline Boards via commercial GigE switches. Next Gen Corr Workshop: EVLA Correlator—Carlson

28. Schedule • Slipped about 1.5 yrs from original 2001 schedule…various reasons. • 10 proto corr chips arrived June 20/06. • Baseline Board…probably August/06…some PCB fab problems have delayed delivery. • Station Board…signed-off design for fab quotation…probably get assembled board late September/06. • Depending on how well testing goes: • 1st prototype corrs at VLA/Jodrell Bank…mid ’07. • Full production mid 2008. Full installation early 2009. • Turn off old correlator end of 2009. Next Gen Corr Workshop: EVLA Correlator—Carlson

29. Summary Response • Type of correlations: all • Output req: spectral + continuum • Special processing req: tunable sub-bands, high spec res, pulsar, 4/7-bit corr, RFI blanking/robust; RFI excision in backend CPUs eventually; possible RT RFI cancel. • Input BW + digitization: 8x2GHz 3-bit; 4x1GHz 8-bit; dedicated fiber I/F on mezzanine card; Other BW/digitization supported; VSI I/O. • #spec channels at max BW: 16k, increases by 2X with each decrease in BW by ½. • #baselines correlated: 528 baselines, 163,840 cross-corrs • Integration times: min ~15 usec with binning. 11 msec standard…scalable with backend and O/P link; unlimited max. • Dynamic range: ~60 dB spectral. • Scale: large, near limits of technology but not “bleading edge”. • Technology: FPGAs, custom hardware, 1 ASIC, COTS CPUs/network, standard form factor. • Scalability: unlimited but high flexibility not necessarily best approach for ‘large N’. 4X baselines possible with new ASIC, same architecture. • Flexibility: high: BW/Nant/Nchan/Nbeam/Nbits; RT, nRT VLBI, auto, connected element, integration/dumping independent of system timing. • Architecture: “WIDAR” (hybrid XF). • RFI mitigation: Nbits, RT blanking, high dynamic range. Next Gen Corr Workshop: EVLA Correlator—Carlson