1 / 49

Level 2 Ocean Salinity L2OS v610 & OTT post-processor

Level 2 Ocean Salinity L2OS v610 & OTT post-processor. 5 July 2013. ARGANS & SMOS L2OS ESL. L2OS v610 status. Significant changes from v600 OTT post-processor OTT running average tracking L1 drift, updated daily in DPGS & reprocessing no more compareTBs! TEC from Stokes 3

tekli
Download Presentation

Level 2 Ocean Salinity L2OS v610 & OTT post-processor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Level 2 Ocean Salinity L2OS v610 & OTT post-processor 5 July 2013 ARGANS & SMOS L2OS ESL

  2. L2OS v610 status • Significant changes from v600 • OTT post-processor • OTT running average tracking L1 drift, updated daily in DPGS & reprocessing • no more compareTBs! • TEC from Stokes 3 • impact on high TEC descending orbits; ignore L1c IRI TEC • Multi-threading • > 2 x speed increase • Simplified slim-line UDP; extended DAP • remove unnecessary flags & fields, make it easy for users to extract good quality data

  3. OTT post-processor Real-time salinity retrieval requires near real-time OTT to track short term drift: while processing L1c, L2OS v610 extracts deltaTBs for South Pacific region and writes them to AUX_DTBXY_. The OTT post-processor runs once per day, reads AUX_DTBXY_ and a set of current deltaTBs (from AUX_DTBCUR), writes a new set of OTTs, and updates AUX_DTBCUR. The L2OS processor always uses the latest set of OTT. L2OS processor OTTs L1c OTT post-processor deltaTBs (DTBXY_) current deltaTBs (DTBCUR) L2OS UDP & DAP • By-products: • daily drift statistics available in AUX_DTBCUR.HDR for analysis & monitoring • other (non-OTT) regions may be specified (in AUX_CNFOSF) for monitoring (output in DTBXY_)

  4. OTT post-processor overview • L2OS v610 • for each region in AUX_CNFOSF • apply region snapshot filter // use snapshot if enough measurements & all within region • if OTT region • apply OTT snapshot filter // ignore if TBs out-of-range, RFI, L1c error • apply OTT grid point filter // not ocean, hi/lo WS, ice • if count(snapshots) > Min_Snapshots & count(gridpoints) > Min_Grid_Points • if OTT region apply OTT measurement filter // ignore sun_point, rfi_tails • deltaTB[region] = use forward model to compute deltaTBs & statistics • write deltaTB & statistics to AUX_DTBXY_ • OTT post-processor • merge all AUX_DTBXY_ from previous day with current AUX_DTBCUR • discard old sets of deltaTBs (last-in/last-out) • compute & write new OTT using current set of deltaTBs • write current set of deltaTBs & statistics to new AUX_DTBCUR

  5. OTT post-processor: L2OS v610 algorithm AUX_DTBXY count[x,y] deltaTB[x,y] stdra[x,y] flags[x,y] Statistics for fov, affov, eaffov, & filtered by flags • for each selected measurement • map xi,eta onto x,y grid • count[x,y] += 1 • sumDeltaTB[x,y] += TBsmos – Tbmodel • sumDeltaRA[x,y] += (TBsmos – Tbmodel) / ra • sumDeltaRA2[x,y] += ((TBsmos – Tbmodel) / ra)2 • flags[x,y] |= selected L1c & L2OS flags • for x = 0 to 128 • for y = 0 to 128 • n = count[x,y] • write n • mean_deltaTB[x,y] = sumDeltaTB[x,y] / n • write mean_deltaTB[x,y] • stdra[x,y] = sqrt(((n * sumDeltaRA2[x,y] – sumDeltaRA[x,y]2) / (n * (n – 1))) • write stdra[x,y] • write flags[x,y] Flags for off-line analysis:

  6. New OTT post-processor section in AUX_CNFOSF Source of geophysical reference parameters for computing deltaTBs: 0 = climatology, L1c TEC & ECMWF priors 1 = model 1 retrieved parameters deltaTB reference max_orbits (nominally 10) quality thresholds (RFI, L1c errors) ott_strategy (mean/gaussian) ott_validity_start (first/mean/last) List of regions Name, ID type (OTT/REG) orbit dir (A/D/?) lat/long snapshot thresholds grid point thresholds And new filters: OTT_SNAPSHOT_FILTER OTT_REGION_FILTER OTT_STATS_FILTER AUX_CNFOSF <OTTPP> section

  7. OTT post-processor components: AUX_DTBXY_ n x regions: ID snapshot start/stop times snapshot start/stop ID Statistics for FOV, AFFOV, EAFFOV, & filtered by flags[x,y]: mean, median, min, max, std. for 3 models for 8 polarisations 4 x statistics count[x,y] deltaTB[x,y] stdra[x,y] flags[x,y] AUX_DTBXY_ datablock

  8. OTT post-processor: OSCOTT algorithm read all deltaTB orbits from AUX_DTBCUR // ignore ottA/D for each AUX_DTBXY_ if region = South Pacific OTT region & good quality read deltaTB for this orbit // keep n ascending & n descending deltaTBs... trim deltaTB // discard oldest first ottA = average(deltaTB, ascending) ottD = average(deltaTB, descending) for model = 1 to 3 write ottA(model) & ottD(model) to AUX_OTT1/2/3 deltaTB += ottA deltaTB += ottD write deltaTB to new AUX_DTBCUR // including ottA/D Options: number of deltaTB orbits (nominally 10+10) mean or weighted mean (nominally mean) filter DTBXY by quality

  9. OTT post-processor components: AUX_DTBXY_ Header contains quality information for each region: Snapshot quality information total snapshots in region, number of snapshots used after filtering number of XX/YY/XXY/YYX snapshots number of snapshots with each of 4 L1c error flags set number of snapshots with TB out-of-range, high std/std_Stokes3/4 Grid point quality information total grid points in region, number of grid points used after filtering number of grid points: too near to land/coast; rain/ice contamination missing ECMWF data; low WS (<3m/s), high WS (>7m/s) Measurement quality information total measurements in region; number of measurements used after filtering number of measurements flagged: as sun point, sun tails, with sun/moon/galactic glint contaminated by RFI (by L1 or L2), L1c error flags Quality information is from forward model 1 only: used by OSCOTT to select ‘good’ sets of deltaTBs.

  10. OTT post-processor components: AUX_DTBXY_ Header contains quality information for each region: Snapshot quality information total snapshots in region, number of snapshots used after filtering number of XX/YY/XXY/YYX snapshots number of snapshots with each of 4 L1c error flags set number of snapshots with TB out-of-range, high std/std_Stokes3/4 Grid point quality information total grid points in region, number of grid points used after filtering number of grid points: too near to land/coast; rain/ice contamination missing ECMWF data; low WS (<3m/s), high WS (>7m/s) Measurement quality information total measurements in region; number of measurements used after filtering number of measurements flagged: as sun point, sun tails, with sun/moon/galactic glint contaminated by RFI (by L1 or L2), L1c error flags Quality information is from forward model 1 only: used by OSCOTT to select ‘good’ sets of deltaTBs.

  11. OTT post-processor components: AUX_DTBCUR m x AUX_DTBXY m x AUX_DTBXY + extra sets: set type (L1c, OTT...) region ID orbit direction (A/D) orbit polarisation (D/F) snapshot start/stop times snapshot start/stop ID L1c filename n x regions: ID snapshot start/stop times snapshot start/stop ID for 3 models for 8 polarisations 4 x statistics count[x,y] deltaTB[x,y] stdra[x,y] flags[x,y] OSCOTT for 3 models for 8 polarisations 4 x statistics count[x,y] deltaTB[x,y] stdra[x,y] flags[x,y] AUX_DTBXY_ datablock AUX_DTBCUR datablock

  12. OTT post-processor components: AUX_DTBCUR Header contains quality information statistics for ascending & descending OTTs generated by the last OTT post-processor run: Ascending OTT quality information count of orbits used to make OTT snapshot start/stop times for 3 polarisations for 8 polarisations 4 x statistics (same format as AUX_DTBXY_ statistics (count, mean, median, std for fov, affov, eaffov, & filtered) Descending OTT quality information (same fields as above) List of L1c datasets

  13. OTT post-processor components: AUX_DTBCUR Header contains quality information statistics for ascending & descending OTTs generated by the last OTT post-processor run: Ascending OTT quality information count of orbits used to make OTT snapshot start/stop times for 3 polarisations for 8 polarisations 4 x statistics (same format as AUX_DTBXY_ statistics (count, mean, median, std for fov, affov, eaffov, & filtered) Descending OTT quality information (same fields as above) List of L1c datasets

  14. OTT post-processor preliminary results deltaTB, XX pol, ascending AUX_OTT1F_ AUX_DTBXY_ x 10

  15. OTT post-processor preliminary results Histograms of deltaTB in XX FOV, ascending orbits, AUX_DTBCUR 20110501 Median = 0.58 K AUX_OTT1F_ AUX_DTBXY_ x 10

  16. OTT post-processor preliminary results std/ra, XX pol, ascending AUX_OTT1F_ AUX_DTBXY_ x 10

  17. OTT post-processor preliminary results Histograms of std/ra in XX FOV, ascending orbits, AUX_DTBCUR 20110501 AUX_OTT1F_ AUX_DTBXY_ x 10

  18. OTT post-processor preliminary results Comparing OTT from DPGS with OTTPP: re-sampled grid XX XX YY YY DPGS OTT (20110504) OSCOTT (20110501) delta = OSCOTT – DPGS Differences along border-belt-suspenders & tails

  19. OTT post-processor issues: good AUX_DTBXY • Which statistics should be used to select/reject DTBXY deltaTB data? • Current implementation has thresholds for maximum % measurements contaminated by L1 RFI, L2 RFI, or with L1c error flags set

  20. OTT post-processor issues: averaging window • What is the optimal running average window? • Averaging many sets of deltaTBs gives better quality OTTs but with a slower response to rapid TB drift

  21. OTT post-processor issues: priors or retrievals? deltaTBs are computed as the difference between forward model TBs and L1c TBs Should we run the forward model using priors (SSS from climatology, SST/WS from ECMWF, TEC from L1c or A3TEC)? deltaTB(priors-model1) std(priors) std(model 1 retrievals)

  22. OTT post-processor issues: xi/eta grid sampling v600 v610 1.0 eta -1.0 0.4 eta -0.7 -1.0 xi 1.0 -0.7 xi 0.7 v600: 129x129 (xi ±1.0, eta ±1.0): sampling interval 0.016 v610: 129x129 (xi ±0.7, eta -0.7 to 0.4): sampling interval 0.011 in xi, 0.008 in eta What grid sampling interval is best for OTTs? Grid point density varies within snapshot – highest at leading edge of AFFOV Typical total 5600 grid points, >100 along leading edge (±0.25 xi): ~0.005 sampling? Blackman pixels ~0.04 diameter: with pixel overlap 1.4, sampling interval 0.028 Small sampling interval leads to higher noise in OTTs.

  23. OTT post-processor issues: merge OTTs? long integration time short integration time delta OTTs are similar – should they be merged?

  24. OTT post-processor issues: merge OTTs? long integration time short integration time OTTs are similar – should they be merged to reduce noise (especially for short integration time)?

  25. OTT post-processor issues: merge OTTs? XXY YYX delta OTTs are similar – should they be merged?

  26. TEC from Stokes 3: descending orbit, high TEC L1cTEC & A3TEC tecu latitude tecu latitude Cross-verification between Matlab breadboard & v610: delta = breadboard-v610 L1PP v600 20110505T040402

  27. TEC from Stokes 3: descending orbit, high TEC L1cTEC & A3TEC tecu Tuning: decreased sigma prior, increased gaussian smoothing, added outlier filtering, TEC height changed from 450 to 400 km latitude L1PP v600 20110505T040402

  28. TEC from Stokes 3: descending orbit, high TEC L1cTEC & A3TEC tecu L2OS retrieved TEC follows A3TEC better than L1c TEC latitude L1PP v600 20110505T040402

  29. TEC from Stokes 3: descending orbit, high TEC L1cTEC & A3TEC L1OP v601 using consolidated VTEC tecu latitude L1OP v601 20110505T040448

  30. TEC from Stokes 3: ascending orbit L1cTEC & A3TEC tecu Ascending orbit with low TEC – noisy A3 TEC retrieval, but good match with L1c latitude L1PP v600 20100802T153815

  31. TEC from Stokes 3: ascending orbit L1cTEC & A3TEC tecu Tuning: decreased sigma prior, increased gaussian smoothing, added outlier filtering, TEC height changed from 450 to 400 km latitude L1PP v600 20100802T153815

  32. TEC from Stokes 3: ascending orbit L1cTEC & A3TEC tecu L2OS retrieved TEC follows A3TEC better than L1c TEC? latitude L1PP v600 20100802T153815

  33. TEC from Stokes 3: ascending orbit L1cTEC & A3TEC L1OP v601 using consolidated VTEC tecu latitude L1OP v601 20100802T153855

  34. TEC from Stokes 3: altitude of TEC Estimated Prior Faraday rotation depends on TEC altitude (not only on TEC amplitude). Tomography using Stokes 3: first analysis shows mean altitude of electronic layer is about 390 km electron density not located at same altitude along the orbit tomography depends on TEC prior

  35. TEC from Stokes 3: altitude of TEC Changing TEC altitude in v610 from 450 to 400km

  36. Multi-threading L2OS • Motivation • run OTT post-processor per-day during reprocessing (same as DPGS) • average L2OS run time in DPGS/reprocessing is 2.5 – 3 hours (TBC) • x3 speed-up with 4 threads would match SM performance (Antonio de la Fuente) • 15% faster CPU clock in reprocessing platform • L2OS maximum run time per half-orbit is ~370 minutes (Pacific Ocean) • (on 5 year old original specification server) • L2OS v610 uses OpenMP to multi-thread time-consuming loops • Pacific Ocean test orbit results (includes 25 minutes I/O) : • 3 threads: 160 minutes (5.3G) – 2.3 x speed-up • 4 threads: 144 minutes (5.1G) – 2.6 x speed-up • average run-time (20110416) with 4 threads 124 minutes (max 166, min 77)

  37. Simplified UDP & extended DAP • Motivation • UDP includes too many flags & fields – confusing • make it easy for users to extract good quality data from UDP • move unnecessary UDP flags & fields to DAP • UDP is for oceanographers, not for analysis • make the DAP easier to use • variable length grid point records in v600 makes reading DAP grid point data very slow • restructure DAP into a block of all grid point data (fixed size records), followed by a block of all measurement data (variable size records) • users/ESL can read just the grid point data & ignore measurements (measurement data can still be read slowly if needed) • Summary • new UDP only contains geophysical outputs from L2OS processor • simplified UDP quality index & quality flags for each model • no compatibility with existing UDP/DAP readers • requires new versions of SDV, GMT, Beam toolbox & bespoke tools

  38. Improving the UDP v600 (176 flags & 57 fields) v610 (16 flags & 29 fields) n x grid points: Grid point data ID, latitude, longitude, footprint, time Science flags (4 x 30) land/sea/coast, rain ice (4 flags: climatology, ECMWF, TBs, Acard) high/low wind/SSS/SST, sea state Geophysical priors WS & SST from ECMWF Geophysical retrievals SSS1/2/3/Acard & sigmas TB42.5 (8) modelled BTs & sigmas at surface & antenna Control flags (4 x 28) range, sigma, chi, chi2, sun/moon/gal glint max iterations, low/min #measurements, too many outliers, convergence limit, poor retrieval, poor geophysical, RFI suspect, RFI prone Product confidence (34) chi, chi2, #iterations, quality for SSS1/2/3/Acard valid, border, affov, tails, sun/moon/gal glint, ice RFI probability counters for DGGRFI x_swath n x grid points: Grid point data ID, latitude, longitude, footprint, time x_swath, distance to coast Science flags (4) near land suspect rain, ice (Acard), RFI Geophysical retrievals (2 x 7) SSS1, SSS2, SSS3, Acard WS, SST, TEC, sigmas on all retrieved parameters Product confidence quality index for SSS1/2/3/Acard Product confidence flags (4 x 3) High/medium/low for SSS1/2/3/Acard RFI probability counters for DGGRFI

  39. Proposed UDP v610 quality flags (per model) • Fg_high_quality • High quality flag is set if Fg_ctrl_poor_retrieval = 0 & Fg_ctrl_poor_geophysical = 0 & Dg_af_fov > 130 (grid points with lots of measurements in the AFFOV). Users filtering with this flag will get the best quality, suitable for cal-val. • Fg_medium_quality • Medium quality flag is set if Fg_ctrl_poor_retrieval = 0 & Fg_ctrl_poor_geophysical = 0. Users filtering with this flag will get all the high quality retrievals, plus those with Dg_af_fov <= 130. This set is suitable for oceanography, , since we know there were no retrieval or geophysical problems. • Fg_low_quality • Low quality flag is set if Fg_ctrl_poor_retrieval = 0. Users filtering with this flag will get all the high & medium quality retrievals, plus those without any retrieval problems (but there may be geophysical problems). This is the lowest quality we would recommend to users: these retrievals have passed the chi2P/max iterations/convergence tests, so may be ok.

  40. Improving DAP readability v600 v610 n x grid points: ID, lat/long priors & sigmas (4 x 14) posts & sigmas (4 x 14) oor flags (4 x 27) fields (8) n x grid points: ID, lat/long priors & sigmas (12) posts & sigmas (24) science flags (4 x 30) control flags (4 x 28) oor flags (4 x 27) fields (45) Fixed block size grid points Variable block size grid points Extra fields & flags from v600 UDP m x measurements: flags (4 x 30) fields deltaTB (x4) TBgal (x2) n x grid points: ID Variable block size measurements m x measurements: snapshot ID xi, eta flags (4 x 30) fields deltaTB (x4) TBgal (x2) v600 priors, posts & sigmas (112 fields) allow for 7 parameters per model – only 5+4+3+2 parameters retrieved: 50% unused. For a large DAP (eg South Pacific) with typically 150k grid points & 120 measurements/grid point: v600 = 360M (70M grid points + 290M measurements) v610 = 400M (50M grid points + 350M measurements)

  41. L1OP v601 & L2OS v610 32 ascending & 25 descending L1OP v601 orbits processed using L2OS v610, deltaTBs extracted from AUX_DTBXY_ Orbits don’t intersect OTT region, so deltaTB statistics extracted for South Pacific test region 122 (30S to 0, 160W to 120W) Some correlation, but: climatology for 122 OTT shifted (late) v601 reduces long term drift?

  42. L1OP v601 & L2OS v610: RFI test orbit 20100709T034234 L1OP v504 REPR L1OP v601 Fm_L1c_RFI set mostly by Fm_RFI_XX & Fm_RFI_YY

  43. L1OP v601 & L2OS v610: RFI test orbit 20100709T034234 L1OP v504 REPR L1OP v601 (using L2OS v600 L1OP v504 REPR OTTs) For RFI (& other) tests we need to ensure OTTs match L1c...

  44. L1OP v504 & v601: RFI test orbit 20100802T153855 L1OP v504 REPR L1OP v601 Region ID 126 - Equatorial Ocean ±10: small islands? >1K delta in mean/median

  45. L1OP v504 & v601: RFI test orbit 20100802T153855 L1OP v504 REPR L1OP v601 delta = v504 – v601 v601 has larger spatial bias than v504 – why?

  46. L1OP v504 & v601: RFI test orbit 20110128T125103 L1OP v504 REPR L1OP v601 Region ID 126 - Equatorial Ocean ±10: RFI @ 5N

  47. L1OP v504 & v601: RFI test orbit 20110128T125103 L1OP v504 REPR L1OP v601 delta = v504 – v601

  48. L1OP v504 & v601: RFI test orbit 20110128T125103 L1OP v504 REPR L1OP v601 Improved Stokes 3 & 4

  49. L2OS v610 status summary • Verification scheduled August/September • requires at least 1 month reprocessed L1c to verify (May 2011) • OTT post-processor – run at ARGANS • TEC from Stokes 3 – run on G-POD? • multi-threading • Remaining work before delivery • Simplified slim-line UDP; extended DAP • need specification of weighted distance-to-coast (TBC) • New galactic map LUTs? • Schedule • delivery mid-September, FAT early October?

More Related