1 / 12

ETA Data Processing

Learn about low-frequency software and techniques for processing RFI data in this workshop. Explore challenges and solutions for observing in the 29-47 MHz range. Discover methods for RFI mitigation and efficient offline processing. This workshop is designed for professionals interested in data analysis and radio frequency interference.

jcave
Download Presentation

ETA Data Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ETA Data Processing Steve EllingsonLow Frequency Software Workshop – Chicago – Aug 10, 2008

  2. RFI Environment: Bad But Manageable ~ 100 s of noise-limited sensitivity using > 95% of contiguous 5 MHz band around 38 MHz KEY POINT: Can observe here – but need good linearity and narrow channelization Often, but not always possible. Search Range (29-47 MHz) Primary threat to linearity – receiver design challenge Ch 6 Ch 5 Ch 3 TV Ch 4 Ch 2

  3. In-Band RFI Challenges Galactic background clearly visible underneath sparse RFI Self-RFI is a relatively minor problem 6-m Amateur Radio Self-Generated (PC) Wideband junk NC State Police Impulsive noise starts to become a problem at resolutions ~100 ms Wideband junk Citizen’s Band, other HF Ionospheric enhancement Ionospheric enhancement Wideband junk

  4. Offline Processing Up to 200 x 1GB (17s) Files 7+7 bit complex @ 7.5 MSPS Data transfer errors (rare but significant) Sample value histograms / clipping (checking for intermittent RFI swamping) Data integrity check 1K FFT (yields freq-time resolution 7.324 kHz x 136.5 ms) Integrate to 8.738 ms (for Crab GP search; also, suppresses impulsive RFI) Create raw spectragrams Updated every ~7.5 minutes (timed to track Galactic background variation) using spectragrams hand-picked for low RFI Create baseline spectragrams Remove frequency response; Linear interpolation between baseline spectragrams to track Galactic background Calibrate spectragrams • Three passes of “plinking” (replacing extreme values with median values): • Time-frequency pixels one at a time [th1] • All freq pixels for a given time, triggered on total power thresholding [th2] • All time pixels for a given freq, triggered on integrated spectrum thresholding [th3] RFI mitigation Operates on 7.324 kHz x 8.738 ms spectragrams w/o interpolation Incoherent dedispersion In effect, smoothing to expected resolution of scattered-broadened pulse (We use 498 ms for Crab) Integrate time series Difficult to automate due to RFI and time-domain baseline fluxuations Manual inspection for pulses Possible Incoherent combining of polarizations / dipole signals

  5. Example of RFI Mitigation Before Dn = 7.324 kHz Dt = 498 ms 3.75 MHz 38.0 MHz Plotting power; Extreme values in this plot are typically within a few % of mean 3600 s After th1 = 0.40 (time-freq) th2 = 0.03 (time) th3 = 0.02 (freq) < 1% pixels plinked Dn = 7.324 kHz Dt = 498 ms 3.75 MHz 38.0 MHz

  6. Example Simple Pulse Detection (old toolchain – sorry!) No RFI Mitigation, No Dedispersion 5s RFI Mitigation, No Dedispersion 5s RFI Mitigation, DM = 56.791 pc/cm3 DM sweep Duration ~ 2 s Peak DM = 56.791 pc/cm3 Est. flux ~ 876 Jy

  7. Example of Relatively Good RFI Conditions No RFI Mit, No Dedispersion RFI Mit, No Dedispersion RFI Mit, DM = 56.791 pc/cm3

  8. Off-Line Processing Summary • Data processing • Operates on coherently-sampled voltage data (dipoles or beams) • 1 hour of observation is typically about 1 TB raw (data constipation!) • 100% new C-language source code / tool chains • Nothing special for computing (tend to use existing PC cluster to minimize amount of data transfer) • Lessons Learned(from the perspective of a dispersed pulse hunter) • Value of extensive diagnostic “pre-analysis” to identify problematic data: Smallest fraction of FLOPS, but greatest fraction of person-hours • Weak RFI (histograms over many domains & resolutions) • Spurious ionospheric conditions • Consistency with sky model (“Error” in time-varying continuum small?) • Repeatability (is today within a few tenths of percent of yesterday?) • Seems to be more productive to reobserve than to try to salvage “subtly problematic” data, even if only portions look bad. • By our standards, we end up throwing out about ½ of data that initially looks good • Extent of site multipath (self-inflicted), impact • Antenna & cable dispersion, impact • Value in keeping coherent dipole voltage data, despite logistics, to maximally facilitate reprocessing

  9. ETA A/D-RX Board 120 MHz System Clock Altera Stratix EP1S25 25,560 LEs 80 9-bit DSP blocks 1,944,576 memory bits LVDS direct-connects via Mictor connector Parallel (4b + CLK) LVDS to RCC: 7.5 MSPS I7+Q7, plus in-band data (240 Mb/s) 47 29 18 Analog Signal From ARX 12-bit, 120 MSPS digitization 3.75

  10. Reconfigurable Computing Cluster (RCC) • 16-node “Virtual FPGA” • Each node is a development board with Xilinx XC2VP30 FPGA • Edge nodes (“E”) catch streaming LVDS from digital receivers • 3.125 Gb/s Infiniband-like interconnects • Center nodes (“C”) route between RCC nodes & push results to PC cluster • PPCs internal to FPGAs run Linux, perform GPP-type functions Xilinx ML310

  11. RCC “All Dipoles” Mode 240 MB/s aggregate (60 MB/s per PC) Coherent time series, 3.75 MHz BW

  12. Acknowledgements: John Simonetti Phys Cameron Patterson CpE Zack Boor Phys Sean Cutchins Phys Kshitija Deshpande EE Mahmud Harun EE Mike Kavic Phys Anthony Lee EE Brian Martin CpE Wyatt Taylor EE Vivek Venugopal CpE Pisgah Astronomical Research Institute Supported by: AST- 0504677 http://www.ece.vt.edu/swe/eta

More Related