1 / 21

Status and Plans for the International Comprehensive Ocean-Atmosphere Data Set (ICOADS )

Status and Plans for the International Comprehensive Ocean-Atmosphere Data Set (ICOADS ) . Scott Woodruff (NOAA/ESRL USA) Contributions include: Eric Freeman (NOAA/NCDC USA) Sandy Lubker (NOAA/ESRL USA) Shawn Smith (FSU/COAPS USA ) Clive Wilkinson (UEA, UK) Steve Worley (NCAR USA).

ryder
Download Presentation

Status and Plans for the International Comprehensive Ocean-Atmosphere Data Set (ICOADS )

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Status and Plans for the International Comprehensive Ocean-Atmosphere Data Set (ICOADS) Scott Woodruff (NOAA/ESRL USA) Contributions include: Eric Freeman (NOAA/NCDC USA) Sandy Lubker (NOAA/ESRL USA) Shawn Smith (FSU/COAPS USA) Clive Wilkinson (UEA, UK) Steve Worley (NCAR USA) 4th Atmospheric Reconstructions over the Earth (ACRE) Workshop KNMI, De Bilt, The Netherlands, 21-23 September 2011

  2. Topics RECLAIM (1) Release 2.5 overview (2) Data rescue status (3) ICOADS Value-Added Database (IVAD) (4) Release 2.6.0: planned by Oct. 2012 (5) International data management issues

  3. (1) R2.5 Overview • Major delayed-mode update completed in 2009 • 1662-2007 • Also: “Preliminary” near-real-time updates • Link to Research to Operations (R2O) plans • Data, metadata, and product access • NCAR, NCDC, and ESRL all provide complementary capabilities serving a diverse range of customers • E.g.: ~400 unique users per year just from NCAR • Project web portal: http://icoads.noaa.gov/ • Woodruff, S.D., S.J. Worley, S.J. Lubker, Z. Ji, J.E. Freeman, D.I. Berry, P. Brohan, E.C. Kent, R.W. Reynolds, S.R. Smith, and C. Wilkinson, 2011: ICOADS Release 2.5: Extensions and Enhancements to the Surface Marine Meteorological Archive. Int. J. Climatol., 31, 951-967.

  4. R2.5: Recent platform mixture • Voluntary Observing Ships (VOS) plus drifting and moored buoys, and other marine platform types • VOS metadata (with help from UK NOCS) The official Release 2.5 period (1662-2007) is now extended monthly with “preliminary” real-time data and products based on GTS data WMO Pub. 47 VOS metadata 1966-2007 1955-65 problematic

  5. Post-R2.5 Assessments Data Quality – Part 1:World Ocean Database 2005 AT Errors • Flow of investigation – UK NOCS, NCAR, NOAA NODC • Anticipating a correction in WOD09 • Illustrates the benefits of cross-agency team work and problem solving • Somewhat outside the funding envelop in all organizations Realistic distribution of AT with latitude from WOD Error in one component of WOD Data work takes time and is constantly challenging…

  6. Post-R2.5 Assessments of Data Quality – Part 2:20CR Feedbacks on R2.5 data Filtered by 20CR QC Unfiltered by 20CR QC 155=HSST 197=Danish 701=Maury 702=Norw. ΔP: ensemble mean first guess pressureminus modified obspressure for selected “decks”

  7. RECLAIM (2) Data Rescue Status • Severe downsizing of NOAA/CDMP was a major loss • Can more be done now with Crowdsourcing (& OCR)? • Potential Crowdsourcing issues: • Generating diverse public interest? • Are multiple form types a problem? • Competition for limited capabilities? • Wilkinson, C., S.D. Woodruff, P. Brohan, S. Claesson, E. Freeman, F. Koek, S.J. Lubker, C. Marzin, and D. Wheeler, 2011: RECovery of Logbooks And International Marine Data: The RECLAIM Project. Int. J. Climatol., 31, 968-979. • ICOADS Marine Data Rescue: Status and Future CDMP Priorities(http://icoads.noaa.gov/reclaim/pdf/marine-data-rescue.pdf)

  8. Data Rescue Best Practices:Proposed “pipelining” (a) Imaging • As practical, initiate concurrent processing: • (b) prior to completing (a) • (c) prior to completing (a-b) • Can be helpful to explore data quality/characteristics in advance (e.g. dups) (b) Digitizing (keying) • Resource constraints • All these steps have been costly, but Crowdsourcing is a new (b) option • (c) IMMA translation has generally been under-resourced – but serves as a critical foundation for applications (c) IMMA translation

  9. Green =digitized Yellow=partially Red =undigitized Note: all require translation Current Data Rescue Candidates for BlendingMajor (past) contributions from NOAA/CDMPRECoveryof Logbooks And International Marine Data (RECLAIM) ProjectAtmospheric Circulation Reconstructions over the Earth (ACRE) R2.5 R2.4

  10. US Lightship Collection 1916-82 component; ~430K daily obs. (CDMP-funding project initiated by WHOI) • 33 of 39 files currently available: • undergoing additional QC and IMMA translation • pre-validation of data planned by ICOADS prior to blending

  11. Greenwich Mean Noon (GMN): 1910-47 Obs. (plus many other forms/published data – back to 1870s) • Potential crowdsourcing project? • Imaging 100% complete (~190K pages) and readily available • Very large collection (2.6M rpts)

  12. Additional CDMP-KNMI project possibility for Crowdsourcing: Dutch Extract Journals: 1826-92 • Imaging also 100% complete 17,565 images 193 logbooks from 327 ships • 650K estimated daily reports

  13. German Maury Collection: 1845-67; 544K Due to resource constraints, only an estimated 50% of the collection can be QC’d and translated in next 6-7 months

  14. (3) ICOADS Value-Added Databasehttp://icoads.noaa.gov/ivad/ • Project aim: Address our current inability to trace value-added improvements back to individual ICOADS observationsthrough: • establishment of DBMS to support development of value-added records and facilitate user access; • implementation of supporting modifications to IMMA format • scientifically demonstrate the impact of value-added records on air-sea flux estimates &common climate indicators Limited funding for FY2011-13 obtained from NOAA Climate Program Office Divided among NOAA (ESRL and NCDC), FSU/COAPS, and NCAR E.g. ESRL programmer levels over three years = ~6 person months

  15. IMMA: A Robust and Extensible Observational Data Format . . . core icoads immt meta model suppl • International Maritime Met. Archive (IMMA) format (ASCII) • Core + optional “attachments” • A new IVAD attachment with: • Field number (FN) • Value-added data (VAD) • Author reference code (ARC) • Uncertainty, QC, etc. Key requirement: attm of original data forms: experience demonstrates format translations frequently contain errors or omissions Advantage: exact copy of original permits re-translation and cross-checks at any time

  16. ICOADS Value-Added Database (IVAD): Overview Schematic (1) IMMA core or attms: selected field(FN) e.g. field(35)=SST (2) IMMA IVAD attm: FN, VAD, ARC, etc. Field(35)=SST Value-added data (VAD; e.g. if FN=35 for SST) Author reference code (ARC) (3) ICOADS web service for serving IMMA data, e.g.SST and/or VAD (4) ARC overview table (5) ICOADS-hosted file respository (URL1) (6) Optional external file repository (URL2)

  17. (4) Release 2.6.0 Plans • Data ingest cutoff: ~April 2012 • Available historical inputs – in IMMA format • A variety of data corrections • Will process an updated WOD09 • Incorporate near-surface salinity (+ related data/metadata) • IMMA format improvements in conjunction with IVAD • Other new attachments: • Physical Oceanography (Ocn) • Automated instrumentation (Auto) • Reanalysis feedbacks • to be developed later w/ reanalysis projects • Platform tracking (Track) • storage capacity, but no resources in sight to implement

  18. Establish Unique Report ID (UID)(within Icoadsattm) • 6-digit base-36 number (i.e. alphanumeric) • Tentatively we plan to number the entire R2.6.0 intermediate product (i.e. containing all duplicates): • 1, …, R2.6.0i • Note: R2.5icontains ~295M reports • Subsequent additions would be numbered >R2.6.0i prior to blending (insertion of historical data would disorder the initial pure numeric sequence) • Unresolved questions regarding data report modifications, possible report compositing, etc. Critical advance to provide a permanent ID for each ICOADS record to benefit reanalyses and other applications (plus IVAD work)

  19. (5) InternationalData Management Issues • Proposed network of mirrored WMO-IOC Centres for Marine-met. and Ocean Climatological Data (CMOC) • Proposed requirements: • Host standardized formats and QC processing • Reliably mirror data and products • Open data access; WIS (WMO Information System) interoperability • Benefits e.g. historical data exchange • Countries can be reluctant to exchange historical data without assurance of formal international repository JCOMM Marine Climate Data System (MCDS) Workshop 28 November-2 December 2011, Hamburg Germany

  20. Conclusions • Regular MARCDAT/CLIMAR workshops (~every 2 yr) • data focus; help drive progress & develop shared ownership • latest: MARCDAT-III, 2-6 May 2011, Frascati • Overall “CLIMAR” initiative • http://www.marineclimatology.net/web/ • Involvement with satellite projects and the surface temperature (land) initiative offers an important new avenue for closer linkages between communities • E.g. interoperable tracking of data provenance (UID) • QC and bias-adjustment improvements needed • e.g. static QC limits extensively missing for high-latitude data • link with IVAD work – cross-checking via 20CR results?

  21. Scott.D.Woodruff@noaa.gov

More Related