Ground Systems Data in the World Data Center Archives

1. Ground Systems Data in the World Data Center Archives Justin Mabie (NGDC) and Anatoly Soloviev (GCRAS) 2 July 2011

2. Ground System Data Holdings Solar Ionosonde Indices Magnetometer

3. Old School Data in Climate Controlled Storage Some are neat, some are a mess, some are so old our computers can’t read them. Boxes and folders and files – oh my!

4. In the NGDC Warehouse: In the past, these data have been migrated to film or packed away in boxes. Modern digital data is packed away neatly in our ADIC tape library system, which is cheaper, less labor intensive, and does not require a large climate controlled warehouse. To get space in the warehouse, we spend countless hours bringing the data together on a single medium and creating inventories and metadata records.

5. Geomagnetic Data Ingest System NGDC workflow for geomagnetic observatory data

6. Solar Data Ingest System NGDC workflow for solar observations

7. ISO Metadata Workflow Metadata is a real challenge. This workflow shows how NGDC ingests metadata, performs quality control, ports it to the ISO metadata standard, loads it into databases, and disseminates it to our customers.

8. Ionosonde – User Services Real Time Applications Users Include NWS SWPC USAF Weather Agency Research Lab Northwestern Research Associates Australia Ionospheric Prediction Services Univ. MA, Lowell Center for Atmos. Research Space Environment Technologies

9. GRAILS/ORACLE Databases for Metadata NGDC is developing databases so our users can make use of our metadata in ways we can’t think of. We traditionally focus on the needs of the environmental scientist – what about the instrument and hardware developers, the social scientist, the politicians? For further information, contact: geomag.obs@noaa.gov

10. XML – Metadata Format

11. CDMP – Climate Data Modernization Project CDMP Data Recovery for Magnetometer, Solar, Ionosonde, and Airborne Imagery Example – Geomag: Title: CDMP - Geomagnetism Volume: Approximately 4TB per year of data Storage: ADIC tape library Dissemination: Reduced resolution images will be put on FTP site; full resolution data available by request. • Notes: • Data volume is very large (many terabytes) • Full filenames are included to reduce NGDC overhead (lesson learned from earlier projects) • Holdings range in date from 1957 – 1983 for Geomag. • NGDC film magnetograms were on track for completion in 10 years. Solar data is near completion, and Ionosonde data is aprroximately 10% modernized. STATUS: All efforts De-Funded due to fiscal crisis. Procedure is in place, providing an opportunity for future funding.

12. Geomagnetic and Solar Indices Bulletins Title: Geomagnetic Indices Bulletin Description: Monthly publication containingKp, Ap, Cp, An, As, Am. http://ngdc.noaa.gov/stp/geomag/indices.html Solar Indices bulletin contains Sunspot Number, Sunspot Prediction and Solar Radio Flux data. • DATA SETS • Printed forresearch purposes Media: online only NGDC Holdings: 1996 – Present URL: ngdc.noaa.gov/stp/geomag/geoib.html ftp.ngdc.noaa.gov/STP/SOLAR_DATA/Publications/

13. Geomag Observatory Metadata Form Page 1/5 XML Page Return Page 2/5

14. Coming soon! RSS feeds to notify users of product changes What is RSS? RSS is a way for a user to automatically retrieve information from selected websites and aggregate it. The feeds (in XML format) are collected with an RSS reader. Readers for RSS include email applications such as Mozilla Thunderbird and fancier readers such as Google Reader. Subscribers to RSS feeds maintain their privacy. How will RSS feeds be used at NGDC? Users will select RSS feeds for specific data types. Subscribers will get occasional automatic alerts for: format changes reprocessed data sets new data sets

15. Thank You!