1 / 30

CDAAC Status and Strategy

CDAAC Status and Strategy. Bill Schreiner and CDAAC Team COSMIC Retreat Nov 4, 2010. Outline. CDAAC Objectives CDAAC Overview CDAAC Status Summary of CDAAC mini-review Processing COSMIC Follow-on missions and missions of opportunity. CDAAC Objectives.

cyrah
Download Presentation

CDAAC Status and Strategy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CDAAC Status and Strategy Bill Schreiner and CDAAC Team COSMIC Retreat Nov 4, 2010

  2. Outline • CDAAC Objectives • CDAAC Overview • CDAAC Status • Summary of CDAAC mini-review • Processing COSMIC Follow-on missions and missions of opportunity

  3. CDAAC Objectives • Provide high quality GNSS RO data to the community in NRT, post-processing, re-processing (re-analysis) • Improve data processing algorithms to maximize impact of RO data on research and operations • Support data users, TACC • Conduct scientific investigations when appropriate and efficient

  4. CDAAC Personnel • Program Director: Bill Kuo • Chief Scientist: Chris Rocken • CDAAC Manager: Bill Schreiner • S/W Engineers • Doug Hunt, Lead • Maggie Sleziak-Sallee, Web • Teresa Vanhove, GPS • Jason Lin, SE • Karl Hudnut, System Admin/Engineer • Scientists • Bill Schreiner, GPS Lead • Sergey Sokolovskiy, RO Inversions • Xinan Yue, Ionosphere • Janet Zeng, RO Inversions • Tae-Kwon Wee, 1DVAR

  5. Main CDAAC Functions • RO Payload Operation • Configuration control (firmware and tracking configuration) • Scientific/technical guidance for commanding, operation • Near real-time monitoring, trouble-shooting, and Q/C data analysis • Data Processing and Analysis • Level0 unformatting and QC • GPS ground processing (ZTD, site estimation, clock estimation) • LEO POD, and atmospheric excess phase • Absolute TEC generation • Inversions (neutral atmosphere and ionosphere) • Retrievals (1DVAR) • NWP and correlative data handling • Product QC and analysis

  6. CDAAC Functions (cont) • System Operation and Monitoring • H/W, O/S and NFS filesystem • System fail-overs • CDAAC operations • Input data stream monitoring • RTS downlink • GPS Bit-grabber operation and monitoring • GPS ground data (IGS, NRCan, EUMETSAT, COSMIC sites) • IGS and IGU orbits, clocks and EOP • Bernese configuration • NCEP GFS, ECMWF, radiosonde, ionosonde • Data Management, Dissemination, and Archival • Support Data Users

  7. CDAAC Hardware Active I/O Computer Backup I/O Computer Real-Time Setup • Dual String system • Redundant Input/Output IO-computers • Redundant Real-Time Operational cluster computers (with RAIDS) • Real-Time staging cluster to stage (test) software updates • Backup power and cooling Post-Processing Setup • R&D cluster • Re-Analysis cluster for climate re-analysis • FTP Server with RAID

  8. CDAAC Processing Flow CDAAC File types in RED Atmospheric processing atmPhs atmPrf (S4) Full Spectrum Abel Inversion 1-D Var Moisture Correction LEO data Level 0--level 1 Excess Phase wetPrf bfrPrf POD (Orbits and clocks) Fiducial data Real time Task Scheduling Software Profiles Ionospheric processing ionPhs (L1,L2 amp/phase) ionPrf (EDPs) podTec (TEC) scnLv1 (S4) tipLv1 (Radiance) Excess Phase Processing Combination with other data Abel Inversion

  9. Best Effort Monitoring Monitor the system regularly throughout the day M-F 8am-8pm CDAAC Ops team monitors the system 3 times per day on weekends and holidays Available by email and cell phone CDAAC reliability estimated > 99.5%

  10. CDAAC Data Processing Status • Near Real-Time • COSMIC • SAC-C and C/NOFS • Post-Processing (2007.3200-2009.2650) • GPSMET, GPSMET-AS, CHAMP, SAC-C, GRACE, COSMIC • Will attempt to understand 3 “noisy” CHAMP months (2007.3200) • Re-Processing (2009.2650) • Published new CHAMP re-processing • COSMIC re-processing finished, published soon • Then start re-processing for GPSMET, GRACE, TerraSAR-X, SAC-C, Metop/GRAS, C/NOFS

  11. COSMICDataProduct QC • Output file saved from each process • Hard failures logged in cosmic_badprf table • Most products have BAD flags in files and database • Some product files include error characteristics • LEO Orbit Determination (Post-fit residual test, Orbit overlap test) • Neutral atmospheric excess phase files (SNR and phase noise tests) • Neutral atmospheric profiles (Bending angle noise and bias tests, checks for L2 quality, presence of scintillation, gross checks on N/T/P) • Absolute TEC data arcs (cycle slip detection/correction, outliers) • Electron Density Profiles (under development) • Scintillation Indices (under development) • TIP Night-side Radiances (BAD flag for incomplete integration time, motor not at proper destination, ill-defined sensitivity)

  12. CDAAC 3.0 Status • To be released by late November 2010 • RPM-based install system • Improved data access, website, DB interface, monitoring, configuration management, archiving, documentation • All other packages updated to latest versions: perl, postgres, apache, Bernese, .. • Real-time clock subsystem • Improved excess phase for SD processing • Use of higher resolution real-time and post-processing first guess models (GFS high resolution and ERA-40 interim)

  13. CDAAC Review • Held Oct 27, 2010 • What CDAAC development efforts are required for COSMIC and other missions? How much effort? • How do we know our CDAAC GPS RO data processing is optimal? • Review of algorithms and software internally and via publication • Comparison of results (POD, BA, N, T, WV) with other data processing software and investigate and understand differences (i.e. more detailed GRAS study) • Comparison with correlative data, models?

  14. CDAAC System changes • Better system for managing code for production/research • Simplify addition of new missions names in CDAAC code • Restructure ROAM • Refactor genXphase • Add more test suites • Clean up unused code • Continue to add missions • Improved QC, bad flags, error estimates • Multi-GNSS capability, new observables! • Low latency processing • Better systems management • Need to develop CDAAC Level0-Level1 processing modules • Improved monitoring scripts

  15. GNSS Processing • Add GNSS capability, new observables • Improve NRT ZTD • Compare algorithm with Suominet • Improve NRT GNSS clock estimation • 5-sec GNSS clocks • Improve NRT and PP LEO POD (to 0.1 mm/sec) • Process data from both POD antennas • Compute/Apply PCV maps • Implement SLR processing for validation • Improve excess phase processing • Review algorithms, fix bugs, • 5-sec GNSS clocks • refactor • Perform additional validation studies

  16. Neutral Atmospheric Inversions • Restructure ROAM • Add GNSS capability, new observables • Improve wave optics processing • looking for better filtering approaches • looking for alternative methods (like recently introduced WDF) • Improve bending angle optimization • testing methods with reduced weight of climatology • validation by independent data sources in the stratosphere • Improve 1DVAR retrieval code and documentation • Perform additional validation studies (e.g. integrate ROPP package, RO-Trends+) • Requires investigation and understanding the sources of the differences

  17. atmPhs 1) Mission-dependent; 2-4) Mission-independent. 1) Input positions, velocities, raw phase and amplitude, clim. model. Processing removal of NDM, connection of the phase, down-sampling to one rate. Output positions, velocities, connected phase, amplitude, HSL, lat, lon, height of TP. 2) Input output from 1 Processing retrieval of WO (Phase Matching and FSI or CT2) and GO bending angles for L1, L2, (L5) Output GO and WO bending angles for L1,L2,(L5) 3) Input output from 2 Processing ionospheric correction (incl. additional smoothing of L4), connection of GO and WO bending angles. Output ionosphere free connected bending angles 4) Input output from 3, clim. model, atm. model. Processing optimization of bending angles, inversion of N,T,P. Output N,T,P. 1) Phase Connection conPhs 2) Bending Angle Generation (WO,GO) benPrf 3) Bending Angle Correction, Connection bcnPrf 4) Bending Angle Optimization, Inversion atmPrf

  18. Comparison of COSMIC retrieved temperatures to ECMWF for Different optimizations of BA 2009.015-045 Current CDAAC approach: - clim. fitted to obs. BA - mixed with weighting function - WF different for different occs. - on average fitted clim. >50% above h~45 km Tested approach: - obs. BA smoothed with window increasing from 1.5 km at h=30 km to 15 km at h=60 km - clim. >50% above h=60 km for all occs.

  19. Ionospheric Processing • Absolute TEC uncertainty under investigation • DCB estimation, Code/phase leveling uncertainty • Electron Density Profiles • Improving EDPs by using information on horizontal gradients with DA • Improve scintillation products • Add GNSS capability, new observables • Perform additional validation studies • With Paul Straus of Aerospace, JPL

  20. 1-2 TECu uncertainty can be ignored for ground based GPS or low elevation LEO GPS observations. But it might have problem if we want to study the plasmasphere using COSMIC sTEC [eg,2006.230-245]

  21. TEC Precision from Collocated Tracks COSMIC (2006.230-245)

  22. Data assimilation retrieval • Comparison of the latitude and altitude Ne and its retrieval error from simulation • LT=13 • Manmade plasma cave disappears in DA retrieval.

  23. Processing NRT data from MOOs at CDAAC • development, testing, monitoring of RTS downlink • decoding and formatting binary level 0 science payload data. Effort depends on payload architecture • system modification, configuration and management • data processing modifications • testing and validation • NRT operations and monitoring, • post-processing, archival, maintenance • program management • Recent experience with C/NOFS and SAC-C valuable

  24. C-2 Operational Data Processing Requirements Reliable and low latency input data streams Primary and Backup DPC (Data Processing Center) Development system (GNSS capable) Staging system to test processing changes Configuration management of systems Communication access between systems (DPCs, SOCC, RTSs). Increased IT Security? 24/7 monitoring of near real-time operations Maintenance and on call technical support Operator Training Secondary Science payload processing Post-Processing

  25. Configuration Management for C-2 • Cfengine controls • Log rotation • Shared file systems • Processing Daemons • Software integrity • Fail-over between • I/Os and processing • clusters • Disk space/cleanup • Syncing Primary DPC Backup DPC Cfengine Configuration Control Machine Staging Single string Development Single string SVN configuration repository Post- Processing

  26. C-2 DPC Options with CDAAC

  27. CDAAC R&D Schedule for C-2

  28. Recent Activities • June 30th 2010 meeting with NOAA/OSDPD (Kathy Kelly) • Review of GPS RO processing at CDAAC • HW/SW data processing architecture at NSOF • Held additional telecon • Working to test CDAAC within NSOF architecture • Providing info for NOAA CLASS archive • Oct 14th 2010 meeting with AF/Aerospace • Review of CDAAC and SW architecture • Strong interest in using CDAAC to process space weather payloads • Additional meetings in planning • F-7/C-2 JPMO Meeting Dec 7-8, 2010 • Discuss data processing issues

  29. Recent RO Research and Science Apps • Investigated structural uncertainty of RO in LT. Published in JGR • Comparing different methods of estimation of the ABL depth from RO. Investigating global variability. Publishing results soon. • Detection of the sporadic E clouds from RO. In press at GRL • Investigation of the response of the UTLS in terms of gravity waves and tidal variations from COSMIC data to the sudden stratospheric warming event in 2009 • COSMIC EDPs used to estimate ionosphere High Transition Heights (HTH). In press at JASTP • Investigated global ionospheric response observed by COSMIC Satellites during the January 2009 Stratospheric Sudden Warming Event. In press at JGR-SP • Submitted data assimilation paper to improve EDPs. In revision at JGR

  30. Recent RO Research and Science Apps • Global Evaluation of Radiosonde Water Vapor with RO data, Published in J of Remote Sensing • Evaluation of AMSU and AIRS with RO • Impacts of GPS radio occultation data assimilation on regional weather predictions. Published in GPS Solutions • Working on Round-2 of RO-Trends inter-agency comparison • UCAR funded by EUMETSAT to evaluate Metop/GRAS RO data. Final Report due in early Nov

More Related