Lessons from CASPER Correlators at GMRT

Lessons from CASPER Correlators at GMRT Sandeep C. Chaudhari Back-End Development Group GMRT

CASPER Back-End Development Team Sandeep C. Chaudhari Shelton Gnanraj MekhalaMuley KaushalBuch Harshvardhan Reddy Sanjay Kudale IrappaHalagali IndukumarBhonde Ajithkumar B. Yashwant Gupta

Upgrade GMRT Digital Back-End Specifications Number of stations : 32 Max instantaneous BW : 400 MHz Number of spectral channels : 4-8 K Number of input polarizations : 2 Full Stokes capability Dump times : 1 sec or better Coarse and fine delay tracking : (+/-128uS) Fringe rotation : up to 5Hz Walsh switching Subarray support Incoherent and phased array beams (for pulsar work)

CASPER Hardwares at GMRT • SERENDIP-V :- • Developed a 4 antenna single polcorrelator. • Implemented and tested coarse delay and fine delay for it’s functionality. • Provided a Microblaze control of delay correction to the design over UART via cPCI back plane.

CASPER Hardwares at GMRT • iBOB/iADC :- • Developed a 300MHz Wide band pocket correlator for iBOB and tested with antenna signals. • Proposed digital back-end for 15-meter dish at NCRA. • Used as a packetizer in GPU correlator. (SHR/GSJ)

CASPER Hardwares at GMRT • ROACH:- • Ported iBOB based Pocket Correlator to ROACH • Automatic delay and fringe update added. • Tested the design successfully with short and long baselines. • Packetized correlator design modified as per GMRT specifications.

CASPER Hardwares – iBOB/iADC • Developed a Wideband pocket correlator with the help of CASPER h/w & toolflow :- • Wide Bandwidth = 300MHz • Wideband FFT Block = 1024 point real. • Wideband Coarse delay block designed at GMRT added to the design with delay correction up to +/-128uS • Wideband fine delay and fringe stop block designed at GMRT added to the design. • Fractional Delay correction = 1/32 of clock cycle. (11.25 deg over 512 channels) • Max Fringe rate compensation = 32Hz. • Tested the design for it's delay functionality using GMRT antenna signals viz. C02-C06 (baseline ~200m) along with modified 4-antenna design for SERENDIP-V system by GMRT group.

First Fringes with iBOB & SERENDIP-VBaseline ~ 200m & Delay 2.73uS (c. 2008) iBOB – Wideband PoCo First Fringes SERENDIP-V – Correlator First Fringes Uncompensated Delay Cross Spectrum Uncompensated Delay Cross Spectrum Compensated Delay Cross Spectrum Compensated Delay Cross Spectrum

ROACH – A matured CASPER Hardware • Advancements in ROACH :- • A modular hardware with external PowerPC control of FPGA hardware and ethernet connectivity for remote communication that makes the life simpler. • Ample on-chip FPGA resources (BRAM) and onboard memory to accommodate final GMRT upgrade requirements. • A reliable,robust and stable hardware platform with remote hardware monitoring and debugging facility provided. • No JTAG programming required. Hence simultaneous multiboard programming without extra hardware is possible. • MSSGE Toolflow of ROACH board has matured over time.

ROACH based Pocket Correlator IntDly Blocks FractDly & Fstop Blocks

ROACH based Pocket Correlator Cross Phase Poco Norm Cross Amp Cross Phase Cross Phase GSB GHB Norm Cross Amp Norm Cross Amp • Pocket Correlator :- A handy instrument for quick testing of Broadband analog system at GMRT. • iBOB based pocket correlator ported to ROACH hardware. • Automatic delay and fringe update script developed and successfully tested with broadband antenna signals for shorter baselines viz (C04-C11).

ROACH based Pocket Correlator GSB Poco Cross Phase Cross Phase Norm Cross Amp Norm Cross Amp The design successfully tested with broadband antenna signals for long baselines viz (C06-E06) approx.(13.2km)……!!!

Packetized Correlator built with ROACHES • Packetized Correlator for GMRT:- A collaborative efforts of GMRT and KAT-SKA,SA group, in which a 4 antenna dual polarization generic Packetized correlator is developed by KAT-SKA,SA group and we modified according to GMRT upgrade requirements. • Adopted XAUI based design of packetized correlator over 10Gbe approach. • Changes as per GMRT requirements :- • Changed from 2-Antenna 2 pols to 1-Antenna two pols per F-engine. • Delay depth modified from +/-10.25uS to +/- 64uS (expanding to meet longest baseline delay requirements i.e. +/- 128uS) • Corr control package is made compatible with XAUI design. • DEVICE UTILIZATION :- • Number of DSP48Es 269 out of 640 42% • Number of BRAM 80 out of 244 32% • Number of SLICE 25035 out of 58880 42% • It is possible to fit +/-128uS delay depth requirement of GMRT for longest baselines and may be even 4k point FFT.

Packetized Correlator using ROACHES • Lab test procedure developed and extensively tested with noise source. • Long term stability tests carried out. • Delay Tests for it’s functionality– • Integer, fractional delay tested independently & then in combination with an automatic delay incremental loop. • Tested for extreme boundaries. • Fringe block functionality tests :- • Initial theta and fringe rate tested independently and then in combination with an automatic theta incremental loop. • Systematic lab tests results with noise source can be seen here.

Design flow & Test Setup at GMRT Ant Broadband s/g from Optical Fiber Data Tx over 10/100Mbps link Data Dump in GMRT tax analysis prog format FILE ROACH-I F-Engine ROACH-I 2 X-Engines 10Gbe Switch Broadband Signal Conversion & Amplification ROACH-I 2 X-Engines ROACH-I F-Engine Corr ctrl & data acq PC Automatic Dly update prog using astro_cal Noise Source(Pin typ -13 to -15dBm) BW = 200MHz XAUI Link Data Analysis using GMRT tax prog CorrCtrl over katcp Signal Generator (CLK) Pin = 0dBm Fs = 800MHz 4-station Correlator F-engine : 4 ROACHES X-engine : 4 ROACHES Locked to NTP Server 1 PPS sync signal From10MHz Ref

Lab/Antenna Test Setup at GMRT Analog Signal Conditioning Setup Packetized Corr Setup Common Test Setup for FPGA-CPU/GPU-Packetized, PoCo

Packetized Correlator using ROACHES - Long Term Stability • Long term stability test results with noise source. T= 3 hrs Cross Phase Norm Cross Amp

Packetized Correlator using ROACHES Design then successfully tested with broadband antenna signals with short and medium long baselines viz. C06-C11 and C06-S02 respectively. Frequency Band = 1280MHz @BW = 120MHz Frequency Down Conversion done at Baseband frequency.

Packetized Correlator Results Packetized corr Time series Plot • Short baseline tests : • Ants : C06-C11 (baseline ~500m) • Source : 3c286 Cross Phase Norm Cross Amp Packetized corr self spectrum C11 Ant Packetized corr CX spectrum Cross Phase C06 Ant Cross Amp

Packetized Correlator Results • Long baseline tests : • Ants : C06-S02 (baseline ~4.3km) • Source : 3c286 GSB Time series Plot Packetized corr Time series Plot Cross Phase Cross Phase Norm Cross Amp Norm Cross Amp

iBOB as a DBE for 15-m dish Cross Phase • Long term stability issues :- • Normalized cross amplitude fluctuation over time observed. • Asynchronous communication between iBOB and data acquisition PC. • Caused due to corruption of self/cross data while reading by PowerPC & writing by FPGA to shared BRAM. • Long term stability issue solved :- • Provided internal integration counter, that is polled by acquisition program to ensure proper time stamped data reading. Norm Cross Amp Cross Phase Norm Cross Amp

Further Scope Packetized Correlator design need to be expanded to 8, 16 and then 32-station correlator in step by step manner. Existing Delay compensation range of +/-64uS to be expanded to +/-128uS Spectral resolution to be made better than 1MHz. Repeatability tests need to be carried out.

Lessons GMRT group has successfully built CASPER instruments like simple spectrometers and correlator to more complex systems as per GMRT DBE requirements and successfully tested using GMRT antennas for short and long baselines. Gained enough experience in designing, debugging and testing of CASPER over a course time that can be used while building next generation DBE for GMRT. Still miles to go……. A Scope for learning and contributing to CASPER.

THANK YOU!

Lessons from CASPER Correlators at GMRT