140 likes | 258 Views
Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011. EOVSA Technical Design Meeting - NJIT. Digital Packaging Processor. Role of DPP Overall assumptions and priorities Interface Overview Tasks and algorithms Hardware/software Implementation
E N D
Digital Packaging Processor - OverviewGordon HurfordNov 7, 2011 EOVSA Technical Design Meeting - NJIT
Digital Packaging Processor • Role of DPP • Overall assumptions and priorities • Interface Overview • Tasks and algorithms • Hardware/software Implementation ----------------------------------------------------------- • DPP Interface Details
Data System Approach Fundamental Drivers • Very limited software resources • Non-trivial data rate and volume • Automated analysis pipeline for efficient observing • Must have science-useable system in place by September 2013 • Data products to be readily useable by broader solar community • Data products with preset parameters • Data products with user-selected parameters • Tools and support for experienced users
Data System Approach Implications • Monomode observing • Calibrated data archived in application-specific databases • Reliance on existing software packages • Miriad package for calibration & mapping • RHESSI SolarSoft package for user interface and data product display • RHESSI database model • Err on side of over-rejection of data • Limited initial support for ‘nice-to-have’ options • Limited initial support for calibration refinements • Limited support for non-solar applications
Data System Assumptions • All information required for data analysis is written by the DPP to the Interim Data Base • Engineering data acquisition, archiving and display is the responsibility of the ACC, and is “largely” decoupled from science data.
Nomenclature • Data frame = Interval representing data from one correlator cycle (20 ms, ~4000 channels with 500 MHz range) • Spectral frame = Data corresponding to a complete frequency-agile cycle (nominal 1 second, 10s to 100’s of ‘science channels, 18 GHz rang) Corresponds to a state frame. • Scan: Observing interval within which target and frequency cycling pattern is unchanged
Role of Digital Packaging Processor • To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base • Real time, irreversible processing
DPP Interface Overview Interim Data Base DPP <P>, <P2>, Correlations Correlator Start / End Scan Commands Scan-independent Calibration Parameters Miriad format ACC Scan Parameters Frame parameters State Frame Frame status report Internal RFI Database RFI results 1 s timing tick 0.02 s timing tick
DPP Task Timing • Occasional – non operational • Accept, store and preprocess calibration parameters • Scan initiation • Accept, store and preprocess scan-specific parameters • Data frame (20 ms) • filter, and frequency-average correlator output • Spectral frame (1 s) • Assemble, pre-calibrate, reformat and write data to Interim database • TBD • Format results and write to RFI database
DPP – Stage 1 ProcessingEvery data frame (20ms) • Evaluate kurtosis data to identify RFI-affected subbands as a function of frequency only. • Save RFI statistics • Combine with pre-flagged subbands to generate a “destination vector” for each subband • Apply complex gains at subband level ??? • Average subband data into spectral channels • Save 1st 3 moments of averages ???
DPP Stage 2 ProcessingEvery spectral frame (1s) • Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags • Apply time-independent complex gains if available • Apply baseline corrections • Apply non-linearity corrections • Correct for attenuator settings • Correct for spectral simultaneity • Miriad format this is no longer optional • Convert visibility, uv and analysis-relevant state-frame data to Miriad-compatible format • Write spectral frame to IDB • Report DPP status to state frame
DPP - Implementation • Original concept was to follow FASR plan for a cluster-based DPP • Estimate processing requirements for EOVSA DPP at ~100 MIPS = 1/60 of FASR requirements • Implementation will be based on a single multi-core machine • Software organization will be compatible with migration to a cluster if necessary
DPP Software Architecture RFI database IDB ACC State Frame Correlator DPP Coordination Task I/O, data assembly, no processing per se C1 Pointers within shared memory ParameterProcessing Header Processing Stage 1 Processing Stage 2 Processing C2 C2 C3, C4 C2 Conventional, time-independent processing tasks Cn = core within a quad core processor or nodes in a cluster
DPP Status • Software architecture and tasks identified • Detailed definition of interfaces is underway • EOVSA to Miriad format conversion being tested with FST data • (Fortran 77 for Miriad compatibility) • Next: 1. Complete definition of interfaces 2. Code Stage 1 tasks (GH) • Evaluate timing requirements • Code Coordination task (JM). • Detailed definition of processing algorithms • Code of Stage 2 tasks 5. Machine selection and purchase • Development platform? • Goal: Functional DPP to support prototype testing