NOAA’s Climate Database Modernization Program- Data Rescue Activities

1. NOAA’s Climate Database Modernization Program- Data Rescue Activities Program Manager- Tom Ross

2. ← The Climate Database Modernization Program (CDMP) goal is to preserve and make major climate and environmental data available via the World Wide Web The CDMP supports NOAA’s data stewardship through all of its activities Climate Database Modernization Program

3. Climate Data Base ModernizationOutcomes & Benefits to the Nation • Outcomes • Improved access and digital conversion of data stored on aging paper, film & obsolete digital media • Provides partnerships and jobs through private sector companies in 4 states • Benefits to the Nation • Human Dimension:Reducedloss of life & impacts due to environmental events • Business Dimension:Enhanced opportunities and socio-economic gains by considering climate information in decisions • Policy Dimension:More effective government policy by appropriately using climate information in decisions Tenth year of the program (began 2000)

4. Climate Database Modernization Program: 2000-2009 Over 11 terabytes of climate data now digitized • 53 million weather and environmental images online • Hundreds of millions of records digitized now online • International data access and rescue activities • 86 current NOAA climate/environmental rescue projects Imaged Records Example: Glacial Pairs – Muir Glacier, Alaska July 1st, 1842 hourly weather data from Washington, DC, imaged and digitized through the CDMP Program 2004 1941 Keying and Imaging the data increases data accessibility and data integration- Work must be done by CDMP contractors in KY, MD, WV

5. The proposals judged against the following criteria: 1. Supports NOAA’s Strategic Goals 2. Contribution to improved data access & rescue 3. Value to climate community 4 General merit to overall program 5. Cost effectiveness 6. Ease of digitization by the contractors Call for papers- Issued Each July/August Data Access Workshop – Held November each Year 60 Proposals were submitted for FY 2009, 53 in 2006 but delayed until FY 2008 funding CDMP Proposal Process has NOAA Working Together on Data Rescue and Recovery Climate Database Modernization Program

6. NCDC Archives Paper Archives Room

7. IMC handles paper scanning, COOP scanning from microfiche, and priority scanning HOV Services handles bound records, publications, and fragile records SourceCorp handles incoming records, COOP/SAO keying projects CDMP Digitization Partners

8. Non-Digital Data Archive Percent digitized (Keyed or imaged) 50% (55 million) 1.7% (2,105 reels) 1.0% (8,600 fiche) Manuscript / Autograph* 55 Million Pages stored in 120,000 boxes * Located at Asheville; additional paper records located at the Federal Records Center in Georgia that will be inventoried and prioritized for digitization 35mm & 16mm Film 125,129 Rolls Microfiche 860K fiche containing 51 million pages

9. Contents of the Archives: Micrographics

10. Fragile, bound document scanning CDMP contractors have specialized equipment to scan difficult media while preserving its physical state.

11. Large Document Scanning In addition to standard document sizes, contractors can also handle large paper records, such as maps and charts.

12. SWO daily forms: 897,961 images 3,592 per day SWO 1001 monthly forms: 19,368 documents 77 per day COOP microfiche: 369,625 images 1,479 per day What’s being digitized at present?“High Volume” Projects, 2009

13. Climate Database Modernization Program Typical Camera and Imaging Setup for International Projects International Activities High tech digital cameras and camera stands are set up in a well lit environment- sponsored by NOAA/NWS Technicians are trained to image the records using the camera equipment they check for quality, exposure and completeness Digital jpeg images are stored in the camera’s hard drive then transferred to the computer Images on the computer are then written to CD-ROM or DVD and sent to NCDC for storage and keying Keying format developed at NCDC based on data captured from host country Data must be keyed in U.S. by CDMP contractors End to end process can take 1-3 years depending on complexity and amount of data to be imaged/keyed A technical leader responsible for each step in the process at each international location is critical !

14. Climate Database Modernization Program NOAA & the Uruguayan Meteorological Service have entered into a cooperative project on data rescue In coordination with the U.S. Embassy.The U.S. National Oceanic and Atmospheric Administration (NOAA) donated digital equipment used for the project. International Activities-Uruguay and Chile 2004-2008 72,000 images received from the Uruguayan Meteorological Service and the Uruguayan Navy 7 synoptic stations (4 civilian and 3 military) keyed with observations as early as the 1930’s up to 2005 for a total of 9 million keystrokes. Total imaging/keying for on-going project involves data from over 30+ stations totaling nearly 1,000,000 records Keying data for Punta Arenas, Chile data record begins in 1896 Integrate data into NCDC global databases

15. CDMP African Upper Air (Pibal) Project

16. The index on the CD-ROM will be the main point for searching for related data. The index page provides links to all available data types (both images of recorded data and actual keyed data) associated with the CD-ROM Making Data User Friendly and Accessible • Station:67423 Day/Mo/Year Time:07/11/1970 04Z • Keyed Image ID: CDMP06MA\MA0001\0009.jpg • Source Image ID: .\MA0001\1\P6160006.jpg • COMPUTED OBSERVED • TIME HGHT Azimuth Elev DIR SPD DIR SPD • MINS FEET DEG DEG DEG KTS DEG KTS • > 0 0 130.0 5.0 • 1 500 327.2 21.2 147 13 0 0 • 2 1000 337.9 22.4 170 12 0 0 • 3 1500 239.7 24.9 165 8 0 0 • 4 2000 342.7 26.1 174 9 0 0 • 5 2500 337.4 25.5 140 12 0 0 • Note: > Marks the Beginning of each Record in the Launch • The highlighted .jpg has a link connecting it to the original image from which the data was keyed • The data above is an upper air pibal, due to the limited wind data available a keying format was developed to key the azimuth/elevation data along with the surface wind data, found in the header section. The wind data, to the middle right of the image, we found, was often determined by the average of multiple levels of wind data.

17. Climate Database Modernization Program

18. Climate Database Modernization Program One of the earliest forms with detailed hourly data

19. Selected NOAA ProjectsClimate Database Modernization Program Hurricane Celia 1962- Hurricane Wallet Project

20. Electronic Hurricane Wallet Project • Tropical Cyclone 'Storm Wallet' Electronic Archive • After the dissipation of every tropical cyclone occurring in the Atlantic and eastern north Pacific basins, all of the data and relevant materials related to that cyclone are collected by the NHC staff. The materials are placed in a "storm wallet" which currently takes the form of an expandable binder, or series of binders. These storm wallets have proven to be extremely useful in the post-analysis of many tropical cyclones, both near-term and in some cases, decades later. • The procedure for storing this data dates back to well before the routine use of computers in the office environment. In the Atlantic, the wallet series begins in 1958 and proceeds continuously through the present. In the eastern north Pacific, wallets begin in 1988, the year in which operational responsibility for that basin was assumed by NHC. NHC Central Pacific is developing the same “Storm Wallet” archive

21. Electronic Hurricane Wallet ProjectHurricane Camille 1969

22. Typhoon/Hurricane Database Keying task New Delhi Tropical Cyclone Keying Project

23. Typhoon/Hurricane Database Keying task • Tropical cyclone best track position and intensity from New Delhi, India need to be keyed. Photocopies will be provided of data provided by New Delhi. The data are primarily made up of tables of cyclone position and intensity. • Format • The data are requested in a spreadsheet format (e.g., MS Excel). This format was chosen since the data originates as tables. The data should be on one worksheet beginning with the earliest data and working chronologically downward with one blank row between cyclones. The column names are only required to be present once at the top of the worksheet. • Columns • The following columns are requested. They are present in most tables. In the event they are absent, the column should be left blank. It is imperative that the data for each storm have the same columns to facilitate inputting the data into analysis programs. The columns are: • Date • Time (UTC) • Latitude (lat.) • Longitude (lon.) • CI No [or “T. No.”] • Central Pressure (hPa) [or “E.C.P.”] • Maximum Surface Wind (kt) [or “E.M.S.” or “E. Max”] • Pressure Drop (hPa) [ or “Δ P” ] • Grade (text) • Outermost closed isobar (hPa) • Diameter / Size of outermost close isobar • Remarks

24. Typhoon/Hurricane Database Keying task • Caveats • Times should be made consistent as HHMM (e.g., some only report the hour (18) which should be adjusted by zero-filling (1800)). • Landfall is often noted by text in the table taking up an entire line. When keying, this should be moved to the “Remarks” column for the time referenced. • Example: “The system crossed the Orissa coast … around 1500 UTC” should be put in the remarks for the 1500 UTC observation • There is often text following a table describing how the system dissipated. This should be recorded in the Remark column for the last reported position of the storm. • Tables which span a page should be included in one table in the keyed file. • Table numbers and caption are not required. • Damage estimates and other information above the table caption should not be keyed. • In the tables, the date is recorded only on the first observation of the day, with other cells left blank. It is requested that no blank cells be left in the date row, but be filled with the actual date by entering the date in each row. • Systems which are over land do not generally have intensity estimates and are often denoted with “over land” in some columns. In these cases, the text “over land” should be entered in the remarks section with affected cells left blank. • If some columns are missing (e.g., the ΔP column in 1996), it should be left blank.

25. Climate Database Modernization Program International Activities • WWII Royal Navy Ship logs from 1938-1947 were microfilmed and scanned at the British National Archives. The records have been keyed by CDMP and will be QA/QC’d then merged into the International Comprehensive Ocean Atmosphere Data Set (ICOADS). • Expanded to East India Company Logbooks in 2008- Joint UK Met Office Project

26. EIC visualization visualization by Philip Brohan, UK Met Office - Hadley Center http://imma.appspot.com/interfaces/imma_edit_test.html

27. Voyage of the EIC ‘William Fairlie’ from the UK to India to China and back to UK, 1824-1825

28. Voyage of the EIC ‘Windham’ from the UK to China and back to UK, 1816-1817

29. Voyage of the EIC ‘Astell’ from the UK to India and back to UK, 1824

30. Remarks from voyage of the EIC ‘Thomas Coutts’ 1818. Although the remarks are in regards to a nasty winter North Atlantic extra-tropical cyclone, additional logbooks from this collection may contain similar entries that may give tropical cyclone information for the Indian Ocean (from remarks and/or from barometric pressures).Notice the pressure from the Barraud’s barometer increasing 0.89” (30+ mb) in 4 hours.

31. The Foreign Data Library The Vast Unknown…

32. Both recent and archived publications are well-indexed The problem is with the actual data in the library Larry Nicodemus is working to create a better inventory and also to identify data that should be keyed The Foreign Data LibraryThe Vast Unknown…

33. The best index we have for our foreign data: The Foreign Data LibraryThe Vast Unknown…

34. Current efforts Surface weather observations Cooperative observations Tropical Publications-Based on Conference Input ! Next few years Publications for customer service and IPS Charts Historic journals Down the road Barograms, thermograms, radar Saving data only available on film Summary

35. Climate Database Modernization Program • Password secure system access to Data Cabinets with various data (observations, pubs, climate journals, manuals) • Login http://edads.imcwv.com/default.aspx • e-mail: cynthia.b.karl@noaa.gov for login and password • Mention ‘Tropical Cyclone Project” • Other NOAA Web Sites (e.g. NOAA Central Library)

36. Climate Database Modernization Program • Questions ? • Contact Info • NOAA/ Climate Database Modernization Program • 151 Patton Avenue- Room 516-D • Asheville, NC 28801 • Tom Ross- Program Manager • 828 271 4499 • tom.ross@noaa.gov • http://www.ncdc.noaa.gov/oa/climate/cdmp/cdmp.html