Considerations in the Observation of Weather

1. Considerations in the Observation of Weather Sebastian TorresCIMMS/NSSL

2. Errors of estimates Weather signal decorrelation Acquisition time Ambiguities Range ambiguities Velocity ambiguities Antenna rotation Antenna sidelobes Artifacts Hardware imperfections Quantization and saturation noise Amplitude and phase imbalances Phase jitter Strong point targets Ground clutter Biological scatterers Non-stationary signals Spectral moments Polarimetric variables Limitations in Radar Observations Goal of Weather Signal ProcessingMaximize meteorologically relevant information that can be extracted from radar returns

3. Complex relations and usually not implemented in real-time Coherency Thresholding for Data Quality • Need to flag data that may be corrupted or of bad quality • Only use accurate parameters for further data processing • Clarity of presentation of the meteorological variables is important to users • Criteria for thresholding (a.k.a. data censoring) • SNR • Spectrum width • CSR • SQI (i.e., lag-1 correlation coefficient) • Geometric criteria (e.g., storm tops, speckle) • Statistical criteria (e.g., local continuity) • There is a trade-off between data quality and obscuration

4. 2 dB SNR threshold SNR Censoring in the NEXRAD network • Range gates with non-significant powers are censored • Non significant returns have a SNR below a user-defined threshold • The system allows a different threshold for each moment

5. Overlaid Echo Censoring in the NEXRAD network • Range gates that have significant returns but an unrecoverable velocity are censored • Velocity is unrecoverable due to overlaid echoes in the short PRT • The strong trip can be recovered if the overlay is weak (Ps/Pw > 10 dB) • The weak can never be recovered

6. Part I Methods to Mitigate Range and Velocity Ambiguities

7. nth pulse (n+1)th pulse Ts time Apparent delay < Ts True delay > Ts Maximum Unambiguous Range • If targets are located beyond cTs/2, their echoes from the n-th transmitted pulse are received after the (n+1)-st pulse is transmitted. Thus, they appear to be closer to the radar than they really are! • This is known as range folding • Maximum unambiguous range: ra = cTs/2 • Echoes between ra and 2ra are called 2nd trip echoes, echoes between 2ra and 3ra are called 3rd trip echoes, etc

8. Maximum Unambiguous Velocity • A pulsed Doppler radar measures radial Doppler velocity by keeping track of phase changes between samples that are Ts (pulse repetition time) apart • Recall that the phase shift is f = -4pr/l . Then, the phase change from pulse to pulse is Df = -4pDr/l = -4pvrTs/l • Note that only phase changes between –p and p can be unambiguously detected • Maximum unambiguous velocity • 4pvaTs/l = p va = l/4Ts • This is related to the Nyquist sampling theorem: Doppler velocities outside the ±va interval will be aliased!

9. Range and Velocity Ambiguities on Pulsed Weather Radars • Maximum unambiguous range • ra = cTs/2 • Maximum unambiguous Doppler velocity • va = l/(4Ts) • The Doppler Dilemma: rava = cl/8 • Insufficient to observe severe convective storms at practical wavelengths • NEXRAD specifications:l= 10 cmra = 230 km • This problem is worse for smaller wavelengths! va ≈ 16 m s-1

10. What kind of velocities do we expect to measure? To measure these velocities unambiguously we need a Nyquist interval of at least ±50 m/s For the WSR-88D (l = 10 cm): …and we would have to deal with overlaid echoes!

11. Overlaid echoes Ambiguous echoes Range Ambiguities Ts = 3.1 ms and ra = 466 km Ts = 780 ms and ra = 117 km

12. Velocity aliasing Velocity Ambiguities Ts = 3.1 ms and va = 8.9 m/s Ts = 1.167 ms and va = 23.75 m/s

13. Another PRT Trade-Off • Correlation of pairs: • This is sometimes called Signal Quality Index • It’s a measure of signal coherency • Accurate measurement of power requires long PRTs • More independent samples (low coherency) • Accurate measurement of velocity requires short PRTs • High correlation between pairs (high coherency)

14. ^ ^ Signal Coherency • How large a Ts can we pick? • Recall: • Correlated pairs: • Spectrum width much smaller than unambiguous velocity interval • Increasing Ts decreases correlation exponentially • var(v) and var(sv) increase exponentially also! • Pick a threshold: • Violation of this condition results in very large errors of estimates!

15. 8 m/s 150 km Signal Coherency and Ambiguities • Range and velocity dilemma: rava=cl/8 • Signal coherency: sv<va /p • ra constraint: • This is a more basic constraint on radar parameters than the first equation above • Then, sv and not va imposes a basiclimitation on Doppler weather radars • Example: Severe storms have a median sv ~ 4 m/s and 10% of the time sv > 8 m/s. If we want accurate velocity estimates 90% of the time with an S-bandradar (l = 10 cm); then, ra ≤ 150 km. This will likely result in range ambiguities

16. Range and Velocity Ambiguities • Short PRTs are needed to maintain signal coherency and provide acceptable velocity aliasing • Dilemma: rava = cl/8 • Given l, we can pick va to satisfy our needs. Then, ra is fixed, and it is usually so small that there can be 2nd and even 3rd trip overlaid echoes. • Goal: Reduce obscuration from overlaid echoes(a.k.a.“purple haze”)

17. longPRT 270 km 270 km 170 km 70 km 170 km 70 km 150 km 150 km range overlaidechoes shortPRT 300 km Overlaid indication 2nd trip Overlaid indication 1st trip 50 km range 100 km Legacy R/V Ambiguity Mitigation in the NEXRAD network • Long PRTs are used to estimate powers (reflectivity) and short PRTs to estimate velocity • Long-PRT powers are used to unfold short-PRT velocities • Range unambiguous powers from the long PRT tell us where the echoes come from in the short PRT • Overlaid echoes with comparable strengths cannot be resolved!

18. Legacy R/V Ambiguity Mitigation in the NEXRAD network • Split cut at low elevation angles • Collect two scans at the same elevation angle (one using a long PRT and one using a short PRT) • The long-PRT scan is used to retrieve unambiguous powers and the short-PRT scan to retrieve (range-folded) velocities • Good ground clutter suppression but antenna scans twice at the same elevation • Batch mode at intermediate elevation angles • Collect one scan with interlaced batches of short and long PRTs • The long-PRT scan is used to retrieve unambiguous powers and the short-PRT scan to retrieve (range-folded) velocities • Reduced ground clutter suppression but antenna scans once at each elevation

19. Performance of Legacy R/V Ambiguity Mitigation in the NEXRAD network • Velocity field is obscured by range-overlay censoring (“purple haze” syndrome) • In case of overlaid echoes, only strong-trip velocities are recovered • Strong-trip power must exceed weaker-trips powers by ~10 dB • Velocities from weaker echoes cannot be recovered! Can we do better?

20. Other Techniques to Mitigate R/V Ambiguities • Performance of R/V ambiguity mitigation techniques in the NEXRAD network is inadequate • There are several signal processing techniques for extending the unambiguous range and velocity: • Staggered PRT • Dual PRF • Phase coding • Techniques based on physical modeling (e.g., continuity) are not truly signal processing techniques • Wish list: • Good ground clutter cancellation • Same or improved moment estimate accuracy • Same or improved acquisition time • Comparable computational complexity

21. Can get unambiguous powers up to ra2 Can get unambiguous velocities up to mva1=nva2! R1 R2 R2 R1 T1 T2 T1 T2 … time Staggered PRT Technique • Transmitter alternates two PRTs • Assume T1 < T2 • PRT ratio: k = T1/T2 = m/n (m,n integers) • ra1 = cT1/2, ra2 = cT2/2 • va1 = l/4T1, va2 = l/4T2 • Velocities can be estimated for each PRT • Velocities v1 and v2 alias in different ways • The true velocity can be obtained by using how v1 and v2 alias

22. 3 simple rules Velocity De-Aliasing Algorithm (I) • Algorithm relies on how the short- and long-PRT velocity estimates alias • Example (l = 10 cm) • T1 = 1 ms → va1 = 25 m/s • T2 = 1.5 ms → va2 = 16.67 m/s

23. v1 - v2 True velocity v1 v2 ^ add 2va1 to v1 va2 va1 closest level ^ ^ v1 – v2 Velocity De-Aliasing Algorithm (II) ^ ^ ^ ^ • v1 and v2 are computed from R1 and R2 • A velocity difference transfer function determines the intervals for the different de-aliasing rules • Beware: Estimates have errors Aliased v True v

24. 04/06/03 4:42 GMT KTLXVCP 11 – Batch Mode KOUNStaggered PRT (184 km/276 km) EL = 2.5 deg 148 km 184 km va = 25.4 m s-1 va = 45.2 m s-1 Staggered PRT Performance

25. R2 R1 R1 … T2 T1 T1 time Tornado! Dual PRF Technique • Transmitter alternates two “batches” of PRTs • Batches of pulses are more conducive to better ground clutter filtering • We can use same velocity dealiasing technique as with Staggered PRT • Technique fails with strong shear (i.e., large velocity changes between adjacent batches)

26. Y(0) Y(1) Y(2) Y(3) Y(4) Ts Systematic Phase Coding Technique • Overlaid echoes with no phase coding: • , where K is the number of overlaid trips • Transmitted pulses are phase-modulated with SZ(8/64) switching code • Overlaid echoes with phase coding:

27. Received uncohered signal 1st trip cohered signal Switching code Modulated 2nd trip signal Cohered 1st trip signal Modulation code Undesired term Desired 1st trip autocorrelation Systematic Phase Coding Technique (II) • Cohering for the 1st trip • Two-trip case: • Autocorrelation What kind of Rf would work?

28. This product has to be zero S2(f) Sf(f) S(f) f The SZ(8/64) Code • A contaminant signal with zero lag-one autocorrelation will not bias the pulse pair velocity estimator • a • We need a code with zero lag-one autocorrelation (Chu 1972) • SZ(8/64) modulation code: • SZ(8/64) switching code: • The power spectrum of the SZ(8/64) code is a train of 8 deltas equally spaced in the Nyquist interval • Modulated overlaid trips are evenly spread in the Nyquist interval

29. Strong trip cohered Weak trip modulated v vs Weak trip filtered Weak trip cohered Sidebands vw The Systematic Phase Coding Technique • Transmitted pulses are phase-encoded with SZ(8/64) switching code • Recovery of the weak-trip velocity is not always possible • Phase-coded scan can be preceded by long-PRT scan • Powers from the long-PRT scan are used to determine overlaid echoes in the phase-coded scan

30. 03/03/04 20:28 GMT Legacy Velocity“Split cut” Reflectivity“Split cut” EL = 0.5 deg va = 8.9 m s-1, ra = 466 km va = 28.1 m s-1, ra = 148 km Phase Coding Performance (I)

31. 03/03/04 20:28 GMT Legacy Velocity“Split cut” SZ-2 VelocityShort PRT EL = 0.5 deg va = 35.5 m s-1, ra = 117 km va = 28.1 m s-1, ra = 148 km Phase Coding Performance (II)

32. Staggered PRT scan to retrieve powers up to 300 km and Doppler velocities up to 230 km Uniform PRT (Legacy) 1 scan at each elevation angle Staggered PRT 1 scan at each elevation angle Phase coding (SZ-2)2 scans at each elevation angle Regular long-PRT scan to retrieve powers up to 460 km and phase coded short-PRT scan to retrieve Doppler velocities up to 230 km (operational in 2007) Mitigation Strategy 19.5° 7.0° 1.5° 0.5°

33. Staggered/Dual PRT Non-uniform sampling Complex transmitter timing Spectral processing is difficult Use longer PRTs Good ra = cT2/2 Need velocity de-aliasing algorithm Good va = l/4(T2-T1) Echoes are separated in time Clean recovery Difficult to transfer to operations Phase Coding Uniform sampling Existing transmitter sampling Spectral processing is simple Shorter PRTs Good va = l/4Ts Need separation of overlaid echoes Good ra (can recover two overlaid trips) Echoes are separated in frequency Messy recovery Easier to transfer to operations Staggered PRT vs. Phase Coding

34. Part II Methods to Suppress Artifacts

35. What is Clutter? • Clutter refers to echoes that might interfere with desired signals • Depending on the application, weather signals can be regarded as clutter! • Types of clutter (for weather radars) • Point targets • Ground clutter • Vegetation (seasonal!) • Ground terrain • Man-made structures • Sea clutter • Biological scatterers

36. (Courtesy of Boon Leng Cheong, OU) What does clutter do? • Clutter targets are usually very efficient reflectors of EM energy • Clutter contaminates the signal of interest • Biases reflectivity, Doppler velocity, and spectrum width • This affects all downstream radar products! • Clutter can saturate the radar receiver • Ground clutter can be useful! • It’s always there • Radar calibration • Azimuth • Range • Sensitivity • Refractivity measurements

37. 29 dB airplane birds ground clutter Antenna pattern of the WSR-88D without radome Fighting Clutter (I) • The Antenna • Scatterers seen through antenna sidelobes bias reflectivity, velocity, and spectrum width estimates

38. Point target Effects of Antenna Sidelobes (Courtesy of Khoi Le, OU)

39. Fighting Clutter (II) • The Radar Site

40. (from Billingsley 2002) Fighting Clutter (III) • The Radar Wavelength • Power returned from Rayleigh scatterers goes inversely as the 4th power of the radar wavelength • Power returned from clutter targets (specular reflectors) have a lesser wavelength dependence • Shorter wavelengths offer better clutter-to-signal ratios • The Receiver • Need large dynamic range • Governed by A/D bits (and AGC) and system phase noise • Powers returned from clutter targets can saturate the receiver • Loss of sensitivity

41. This is not a problem for a phased array radar! Effective beamwidth for a scanning antenna as a function of the rotation rate Fighting Clutter (IV) • The Antenna Rotation Rate • Antenna motion combined with pulse-to-pulse processing creates an effective broadened beamwidth • A faster antenna rotation rate results in a larger sc • A phased-array radar has better clutter suppression • The Signal Processor • Stay tuned!

42. Clutter Suppression • Clutter suppression can be done … • On the time-series data (Level I data) • On the moment data (Level II data) • Suppression can be achieved by … • Ignoring gates with clutter (censoring) • Filtering • Range time • Sample time • Time domain • Frequency domain

43. Y(f) X(f) Input Output Filter x(n) h(n) y(n) f f X(f) H(f) Y(f) What is a Filter? • A filter is “a device” that alters the frequency spectrum of signals passing through it • Continuous-time filters have discrete-time equivalents that can be implemented as a computer program or with programmable hardware processors • Time-domain representation: y(n) = x(n)  h(n) • Frequency-domain representation: Y(f) = X(f) H(f)

44. Strong Point Target Suppression • Point targets bias all estimates! • Point targets are easily identified in the Doppler spectrum • How would you remove this artifact? • Interpolation in the frequency domain • Go to the frequency domain • Remove point target • Interpolate through the gap • Go back to the time domain Plane flying toward the radar at 10 m/s

45. P 8x range k k k-2 k-1 k+1 k+1 range gate Strong Point Clutter Suppressionin the legacy NEXRAD network • Spectral processing is not available • Search for discontinuities along range • Range gate k has contamination from point clutter if P(k)>8P(k-2) and P(k)>8P(k+2) • Remove and Interpolate (based on continuity!) • q(k-1) = q(k-2) • q(k+1) = q(k+2) • q(k) = [q(k-1) + q(k+1)]/2 • where qcan be P, v, or sv Actual weather parameters are not recovered!

46. This does not apply to wind turbines or sea clutter! Ground Clutter Suppression • Characteristics of most ground clutter returns • Zero mean Doppler velocity • Narrow spectrum width • Non-zero sc due to: wind, antenna motion, window • In the NEXRAD network sc ~ 0.3 m/s • Ground clutter contamination can be suppressed with a high-pass filter • Time domain • FIR, IIR, regression, matrix, non-linear • Frequency domain • notch, notch & interpolation, spectral fitting • Notch width determination • Depends on the CSR and sc

47. Ground clutter will bias all estimates! Clutter Return Weather Return Unfiltered Doppler spectrum v (m s-1) Notch Width Notch filter frequency response(high-pass filter) Ground clutter residue v (m s-1) Filtered Doppler spectrum v (m s-1) Trade-off! Ground Clutter Filters • Select the filter’s notch width to… • Minimize ground clutter residue • Maximize weather signal retention

48. Weather Return Unfiltered Doppler spectrum v (m s-1) Notch Width Notch filter frequency response Missing weather spectrum will bias all estimates! v (m s-1) Filtered Doppler spectrum v (m s-1) To Filter or not to Filter… • Should we apply the GCF everywhere? • No!

49. Zero isodop Z bias Reflectivity Doppler velocity Stratiform precipitation case from a Great Plains region WSR-88D (Courtesy of Rich Ice, ROC) Filtering everywhere is not the solution! • Ground clutter filters may introduce a bias • Signals with narrow spectrum width and near-zero Doppler velocity are more vulnerable (why?)

50. Can you guess? Reflectivity Filter was: not applied, applied in selected gates, applied everywhere? Doppler velocity