ICOADS: Data, Products, and Access

1. ICOADS: Data, Products, and Access Steven J Worley Scott D. Woodruff Zaihua Ji Richard W. Reynolds

2. Outline • New reference document • Status of Release 2.2 • Access

3. New Reference Document • ICOADS Release 2.1 Data and Products • Worley et al. ,Int. J. Climatol.25: 823-842 (2005) • Special Issue: Advances in Marine Climatology • Copies are available • Release History • 1st, Release 1, 1985 coverage 1854-1979 • 6th (last), Release 2.1, 2003 coverage 1784-2002 • Data Composition • 185 million records • Emphasize record tracking - DCK (Table II), SID (Table III) • Development Combination • Delayed mode (DM) processing - GTS plus other sources • Real-time (RT) mode processing - exclusively GTS

4. New Reference Document Bars = DM Lines = RT • How DM and RT Compare? • Use overlap period 1997  1998 DM decreases the number of drifting buoy observation. MEDS, composites and QA checks. DM increases records for ships, moored buoys, and C-Man

5. New Reference Document Data Usage Considerations • Changing observing systems - temporally decreasing ships since 1989 increasing moored and drifting buoys

6. New Reference Document Data Usage Considerations • Changing observing systems, spatially 45% of possible 2° boxes have 5 samples/month in the 70-80’s Decreasing low counts due to reduced ships observations Increasing high counts due to high frequency sampling from buoys

7. New Reference Document Data Usage Considerations • Other changes • Instrument types and sampling procedures • Codes for measurements and visual observations • Instrument package placement on platforms • Sampling frequency and time of day • More …

8. Status of Release 2.2 • Focus time period, 1998 - 2004 • Reprocessing of 1998-2002 from Release 2.1 • Add on two years, 2003-2004 • Processing Status • Observations • Complete and available from NCAR (12 October) • Further analysis of result to be done • Don’t expect a rerun, but …. • Monthly Summary Statistics (MSG) • Prepare after the observations are fixed (~ 1-2 months) • LMRF (old binary format) will be generated for Release 2.2 • We need LMRF to compute MSG, will transition later • Some past users may want this also - access by special request • Warning: LMRF will be discontinued

9. Status of Release 2.2 • Availability • First from NCAR • All products • Following soon after from ESRL • The former NOAA/CDC is now part of the NOAA Earth System Research Laboratory (ESRL) • Only Monthly Summary Statistics in netCDF format • Future at NCDC • Observations; retrospective and near real-time

10. Status of Release 2.2 • Data Composition • New DM Data, 1998-2004 • TAO/TRITON, PIRATA, JAMSTEC Moored Buoy Research Archives • Including RH » DPT • NDBC Moored Buoy Data • DPT in DM and frequency not known in RT stream • MEDS Drifting Buoy QC’d Archive • WMO Pub 47 metadata (1973 - 2004) • Thanks to Liz Kent and Dave Berry • Kent, et al, 2005: WMO Pub. No. 47 Metadata and an Assessment of Observation Heights in ICOADS. Submitted JAOT • http://www.soc.soton.ac.uk/JRD/MET/eck_pubs.php

11. Status of Release 2.2 • Data Composition • Carry forward RT Data, 1998-2002 • NCEP GTS with DM sources filtered out • New RT Data, 2003-2004 • NCEP GTS with filtered

12. Status of Release 2.2 • Challenges and Misc. Notes • Meshing DM tropical moored buoys • Sampling frequency, hourly and 10-minute • 10-minute reduced to hourly • Clean insert with accurate exclusion of GTS • Daily resolution and matching WMO numbers • Removed buoys with erroneous ID=SHIP from NCEP ship GTS • Eight month period (10/99 - 5/2000) • Fill some gaps in MEDS with NCEP buoy GTS • Made some QA(QC) flag fixes for full period of record • Minor errors transitioning from LMRF to IMMA format • Further checking will be done and documentation prepared • Many details are not told here • www.cdc.noaa.gov/coads/e-doc/other/<dupelim_1998> • Ask questions offline

13. Status of Release 2.2 • Moored buoy coverage • NDBC (approximate) • PMEL/NOAA (TAO and PIRATA) • JAMSTEC • Note: new array beginning in Indian Ocean • European buoys not shown

14. Status of Release 2.2 • Resource issue for US ICOADS team • Current personnel

15. Access • Links to all data and information • www.cdc.noaa.gov/coads (under Data and Metadata) • See Table VI (Worley et al. 2005) t,s,v = subsetting by time, space, and variable

16. Access User profiles from NCAR for the past 14 Months • Who is using ICOADS?

17. Access User profiles from NCAR for the past 14 months • What quantities of ICOADS are requested?

18. Access User profiles from NCAR for the past 14 months • What time domains of ICOADS are requested?

19. Access User profiles from NCAR for the past 14 months • What space domains of ICOADS are requested?

20. Access Mandatory Record Segment IMMA “core”, basic data, 48 fields • Review IMMA structure Optional Record Attachments ICOADS, historical ID (e.g. DCK, SID), QA(QC) flags, etc IMMT-2/FM 13, non-core fields from IMMT and FM-13 In use Ship Metadata, WMO Pub. 47 ship metadata Model QC, e.g. UK model comparison with VOSCLIM To be activated Historical (proposed, primarily for pre-1854 data) Others (e.g. moored/drifting buoys?) Recommended Attachment Supplemental Data – original data in original format

21. Access • New interface to IMMA observations @ NCAR • Put online about two weeks ago • Verify and improve linkages throughout ICOADS portal • Builds on previous successes • Provide ASCII output • Previously from LMRF (binary) now IMMA • All data are available for bulk file download (FTP, Web) • Users select time and space domains • User select how QA(QC) flags are applied • Added services • Access to all attachment data • Auxiliary data sources

22. Access • Highlights from Interface • Filtering application

23. Access • Highlights from Interface • Core variable selection

24. Access • Highlights from Interface • Attachment data – e.g. ship metadata

25. Access • Highlights from Interface • Auxiliary Data – two types • New, likely unique • Replacement, destine to eventually replace other data • Note: Can switch off ICOADS request • Just acquire auxiliary data • If you provide IMMA data we can serve it • Incorporate in ICOADS later New Replacement

26. Access • New interface - Level of automation • Request submission detected automatically by daemon, every 2 minutes • Processing begins immediately, with max. processes = 3 • Processing may take several hours for long temporal requests • IMMA is not efficient for processing speed • Users are notified by email and download via Web or FTP

27. Access • Type of Barometer • Aneroid (AN) • Digital Aneroid (DA) • Length of Vessel (m) • Depth of SST measurements (m) • Height of anemometer (m) • Sample with ship metadata from WMO Pub 47

28. Access • Accessories to go with the data • IMMA documentation • Sample FORTRAN code created dynamically to match data requested • Release ID • Variable definitions

29. Access • Current strategy • Delayed mode processing • File based • Future possibility • DB with core, attachments, position, date • Enable near real-time processing • Processing, O(minutes) • Push or pull data delivery • Could be easily updated (monthly) with GTS

30. Conclusions • New reference document with many details about ICOADS (Worley et al. 2005) • Release 2.2, 1998-2004 (plus other fixes), observations are released to the public. • Monthly Summary Statistics will be computed soon • Broaden access in different forms from the partners soon • Data services are going well (judged from NCAR), especially internationally. • IMMA permits more comprehensive access; to attachment and auxiliary data • Supporting resources in the US are thin. International collaboration has been helpful and more is welcomed. Project recognition from the science community can help sustain support for ICOADS

31. Questions? http://www.cdc.noaa.gov/coads/