Digital Packaging Processor

1. Digital Packaging Processor Gordon Hurford Jim McTiernan EOVSA PDR 15-March-2012

2. 1 1 • Role of Digital Packaging Processor • To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base • Real time, irreversible processing

3. DPP Interface Overview Interim Data Base DPP <P>, <P2>, Correlations Correlator Start / End Scan Commands Scan-independent Calibration Parameters Miriad format ACC Scan Parameters Frame parameters State Frame Frame status report Internal RFI Database RFI results 1 s timing tick 0.02 s timing tick

4. DPP Task Timescales • Data frame (20 ms) • filter, and frequency-average correlator output • Spectral frame (1 s) • Assemble, pre-calibrate, reformat and write data to Interim database • Scan initiation • Accept, store and preprocess scan-specific parameters • Occasional – non operational • Accept, store and preprocess calibration parameters • TBD • Format results and write to RFI database

5. DPP Software Architecture RFI database IDB ACC State Frame Correlator DPP DPP_CONTROL I/O, data assembly, Miriad formatting, no numerical processing C1 Communication via Common Blocks DPP_Process_Parameters DPP_Process_Header DPP_Process_Dataframe DPP_Process _Spectral _Frame C3, C4 …. C2 C2 C2 Conventional, time-independent processing tasks with “Clean” inputs Cn = core within a mullti-core processor or nodes in a cluster

6. DPP_CONTROL-- Jim McTiernan -- • Handles all external I/O • Passes reformated header and calibration data on to processing tasks • Assembles data frame input from correlator and calls processing task only if/when complete • Assembles processed 20ms interval data and calls spectral frame processor only when all accumulations are available • Converts output of spectral frame processor to Miriad format

7. DPP_PROCESS_DATAFRAME • Tasks • Identify RFI-affected subbands as a function of frequency only. • Combine with pre-flagged subbands to generate a “destination vector” for each subband • Save RFI statistics – details TBD • Average subband data into spectral channels • Challenge: 450 MB/s • Each visibility is handled only once • Multiple processors (up to 6) handle successive dataframes • Initial Restriction: • No calibration at the subband level

8. DPP_PROCESS_SPECTRAL_FRAMEEvery spectral frame (1s) • Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags • Apply time-independent complex gains • Correct for attenuator settings • Fine delay corrections ------------------------------------------------------- • Baseline corrections • (not necessary) • Apply non-linearity corrections • Refinement to be added later, if necessary • Correction for spectral simultaneity • not needed if using Miriad wide-band mode • Conversion of visibility, uv and analysis-relevant state-frame data to Miriad-compatible format • May be a included here, in DPP_Control or as a separate Process

9. DPP-Correlator Interface • 4096 x 2 x16^2 8-byte visibilities • 4096 x 2 x 16 8-byte <P2> values •  9 MB / data frame  450 MB/s • Two Dedicated 10 Gb Ethernet link(s) • Interface architecture driven by correlator design

10. Format of Correlator Output • Format of interface based on 3-Feb-2012 EST memo from Nimish, as clarified in 12-March-2012 email • Features: • Accumulation divided into labeled packets, each with an identical header (except for sequence numbers) • Packets divided into ‘chunks’, each of which contain all power, power^2 and correlated data for a given subband • Data for a given subband has both polarizations interspersed, • All non-header data are scaled 4-byte integers • Refinement: • Agreed that correlator output should always correspond to 16-antenna/2 polarizations. • DPP will discard unused antennas/baselines

11. DPP-Correlator Interface Open Issues • Specifics of EST format generalization to EOVSA • Byte order: • Correlator outputs Big-endian (MSB first) • Many machines (e.g. PC’s) use Little-endian (LSB first) • How is the load to be divided between the two Ethernet interfaces? • Current format (and DPP software) assumes sequential input within a given accumulation

12. DPP Status • Overall: • Software architecture and tasks defined • Platform to be acquired • Interfaces: • IDB – Miriad format feasibility demonstrated with EST data • Correlator - as defined by Nimish with a few remaining issues • State frame - will follow protocol defined by GN, details TBD • ACC i – definition at an early stage • Timing ticks – TBD • RFI –TBD • Milestones • Detailed software definition, coding and testing (with EST data) is underway • DPP_CONTROL has successfully associated tasks and processors • EST data loads correctly into DPP input common block • DPP_PROCESS_DATAFRAME well-defined and being coded. • EST to Miriad format conversion has been successfully demonstrated. • Immediate goal: End-to-end testing with EST data

13. Some DPP Open Issues • Do we use spectral or wideband mode in Miriad? • Probably either would work • Initiall implementation will use wide-band mode • (simpler, more flexible but less efficient) • analysis timing tests needed to see if this is an issue • easy to switch to spectral mode • Is GNU Fortran 95 the best compiler choice? • Drawback is lack of support for STRUCTURE statements • Workaround available for DPP_PROCESS_DATAFRAME • Options for software development platforms • Independently develop DPP_CONTROL and processing routines on ‘personal’ machines, then test at OVRO? • Duplicate platform located in Bay area (JM, GH or SSL?) • Duplicate platform at NJIT ? • Need to identify and purchase a multi-processor platform for DPP in near future.