1 / 20

Evolution of Archive Technologies at the National Snow and Ice Data Center Ruth Duerr

Evolution of Archive Technologies at the National Snow and Ice Data Center Ruth Duerr National Snow and Ice Data Center 1540 30th St, Boulder, CO, 80309-0449 Phone 303-735-0136 FAX: 303-492-2468 E-mail: rduerr@nsidc.colorado.edu.

odessa
Download Presentation

Evolution of Archive Technologies at the National Snow and Ice Data Center Ruth Duerr

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evolution of Archive Technologies at the National Snow and Ice Data Center Ruth Duerr National Snow and Ice Data Center 1540 30th St, Boulder, CO, 80309-0449 Phone 303-735-0136 FAX: 303-492-2468 E-mail: rduerr@nsidc.colorado.edu Presented at the THIC Meeting at the National Center for Atmospheric Research, 1850 Table Mesa Drive, Boulder CO 80305-5602 July 19-20, 2005

  2. Outline • A brief history of NSIDC • Current holdings • Current systems • Thoughts on the future Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  3. Outline • A brief history of NSIDC • Current holdings • Current systems • Thoughts on the future Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  4. A brief history of NSIDC • World Data Center for Glaciology • 1957 - U.S. National Committee for the IGY awarded the operation of WDC-A for Glaciology to the American Geographical Society under the direction of Dr. William O. Field • 1970 - WDC for Glaciology transferred to the U.S. Geological Survey in Tacoma, Washington under the direction of Dr. Mark F. Meier • 1976 - WDC for Glaciology transfers to the NOAA Environmental Data and Information Service; an agreement between the University of Colorado and NOAA placed the WDC at CU-Boulder, Colorado under the direction of Roger Barry Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  5. A brief history of NSIDC • National Snow and Ice Data Center • 1982 - NOAA grants NSIDC its name • 1983 - NSIDC receives a grant from NASA to archive Nimbus 7 passive microwave data • 1990 - NSIDC receives funding from NSF for the Arctic System Science (ARCSS) Data Coordination Center (ADCC) • 1993 - NSIDC receives first NASA Distributed Active Archive Center (DAAC) contract Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  6. A brief history of NSIDC • 1996 - Antarctic Data Coordination Center (ADCC) established with NSF support • 1999 - Antarctic Glaciological Data Center (AGDC) established with NSF support • 2002 - Frozen Ground Data Center (FGDC) established with International Arctic Research Center (IARC) support Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  7. Outline • A brief history of NSIDC • Current holdings • Current systems • Thoughts on the future Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  8. Current Holdings • Information Center • ~44,000 monographs, reports, serials, reprints, etc. Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  9. Current Holdings (continued) Analog Archives • ~ 10,000 glacier photos • ~ 7,000 sea ice charts • ~ 1,440 maps • TBD cu ft of manuscripts and other records Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  10. Data Holdings (continued) Digital Archives • ~ 440 publicly advertised data sets • 4.6 million granules in ECS system • > 3.5 million files in non-ECS systems Archive Types • ~ 8 TB on-line • ~ 80 TB near-line • >5 TB off-line • Off-site backups for primary data without recovery agreements Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  11. NSIDC Near-Line Archive Size Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  12. NSIDC Distribution Statistics Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  13. Outline • A brief history of NSIDC • Current holdings • Current systems • Thoughts on the future Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  14. NSIDC Architecture LASP Raytheon (Denver) UCB GE Campus Lan Abilene via Front Range GigaPop GES DAAC (MODIS) AMSR-E SIPS I-SIPS PO.DAAC AMSR L1A EDOS Cisco GE switch ECS Router Ebnet Router UCB non-ECS LAN Firewall • Ingest & Distribution • ftp/file server • tape production • Disk Storage (FC • and SATA @ RAID-5) • Rimage CD/DVD • AMSR L1A PDR/Met • Web Services • Guide Documents • SOTC, All about… • Catalog/DIF generator • Client Interfaces • - EDG,GISMO,PSQ, SNOWI • - ECHO Client (WSRD) ECS Production LAN • Data Server • Manage the archive • -STK Powderhorn w/ • AMASS and ACSLS • Insert data into archive • Search and Retrieve data • from the archive • Ingest Polling Server • ASMR, AMSR-E • NISE • MODIS • ICESat • Infrastructure Services • Email • Calendar Manager • Center-wide Intranet • Backup Services • SNIPS • IDS M&O LAN (Includes M&O Intranet Server and Backup Server) • Data Management • Data Dictionary • V0Gateway • - Inv. Search/Results • Order Requests • - subsetting services • ECHO transfers • Data Pool • On-line storage of most • recently ingested data • (StorNext SAN) • Data accessible via • WEB GUI or ftp • Archive Services • MAID vtl w/ AMASS • (transition from tape • archive in progress) • Off-site archive • Science Processing Dev /Production • SSM/I, NISE, NRTSI • - Std Processing Env. • GISMO/PSQ/WSRD • -backend services • AMSR-E PDR/.Met • ECS Subsetter • Spatial, temporal • and parameter subsetting • of granules (HDF-EOS • Only) • Infrastructure • Backup services • What’s Up • SNIPS • MSS and CSS • Email Gateway • ICESat/GLAS • Remote SCF Server • Visualization • Subsetting Services • (Includes storage for • Archived GLAS products) • HEG Server • HDFEOS-to-GEOTIFF • Order Manager • Manages orders from • V0Gateway, MTMWG, • and Spatial Sub. Server • Transfers orders to PDS • for media requests VJT 7/05

  15. NSIDC Data Catalog • Contains metadata about each published data set • GCMD, FGDC compliant metadata • Used to drive web page creation • Modification underway to include OAIS/PREMIS compliant metadata for all data sets Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  16. NSIDC DAAC ECS Interfaces Metadata and Browse ECHO MODIS L2/L3 Snow/Ice Products [2/T, 1/A]] MODIS L2+ Products [3.6] SCFs QA Updates AMSR-E SIPS [<1/T, <1/A] Data/QA Updates GDAAC ECS NSIDC DAAC NISE Data LDAAC [total <1] LaTIS NISE Data [<1] MODIS L2/L3 Snow/Ice Products MODAPS NISE [120/T, 16/A] NSIDC PDR Server AMSR L1A [1.3] Total Data Distribution AMSR-E L0 Science & GBAD PDSs NASDA 8mm back-up 1st 90 days [1.4] • Aqua AMSR-E Data Products [7] Emergency back-up following L+90 • Aqua MODIS Data Products [16] • Terra MODIS Data Products [40] GBAD PDS [<1] EDOS • ADEOS II AMSR Data Products [1] AMSR-E PDS [1.4] • GLAS Data Products [7] GLAS L0 [5.8] EDOS AQUA Ancillary from EMOS [<1] GLAS L1+ Products [23] ASTER Anc [<1] GLAS SIPS LP DAAC GDAAC QA SCFs SCFs EDG CLIENT MTMGW Subscriptions SCIENCE USERS GENERAL USERS [ ] : Archive Volumes at L+1yr in GB/day T: Terra A: Aqua Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  17. NSIDC/UCB NCEP Global ftp push Reanalysis (CDC) NCEP Global ftp push Reanalysis (UNH) Coriolis SSM/I Wentz Tas FedEx (RSS) Modeling Data rcp (USGS/Denver) Vorticity DVD sidads Dehn Ice Charts ftp pull ARCSS CD ftp push adcc / agdc Glacier Photos Investigators ftp ( arcss ) - SNODAS (NORSC) ftp pull NSIDC/ECS AMSR/AMSR-E L1A, PDR’s and met (ftp push) NISE Data (ftp push) SSM/I TBs ftp pull MODIS Browse (ftp push) (GHRC/UAH rSCF DVD Subsetted GLAS data AVHRR imagery GLAS data (ftp push) (Antarctic Ship) Requests ftp push Sea Ice Charts (Arg) IceCharts/ ftp pull bipolar Reports (NIC) ftp pull IMS (NCDC) (Web server) Requests, results, orders (TCP Sockets) EDG/SNOWI AMSR-E NRT ftp pull (MSFC) Fastcopy GISMO/PSQ glacier AMSR/AMSR-E - kryos push JPL/PO.DAAC Forms GISMO-E Weekly Snow Map ftp pull (Email-server) Documents (Rutgers) Images ftp push/pull Permafrost Investigators Media E - mail and Ftp push html, ftp pull distribution attachments /pull .gif, jpeg, E - mail data, etc. Public User Communities Via UCB Network Via NASA EMSn /ECS ) Media (Tape/CDR NSIDC/UCB Interfaces

  18. COPAN MAID - Early Experiences • Using AMASS to provide a file system front end to a COPAN 200t system configured as a L700 with 7 - 9940 drives • COPAN to act as a drop-in replacement for the STK 9710 • Migration of data underway • The big surprise was performance • Minor surprise is the ever shrinking total archive size Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  19. Outline • A brief history of NSIDC • Current holdings • Current systems • Thoughts on the future Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

  20. Thoughts on the Future • NSIDC DAAC is pushing for an entirely on-line archive • What technologies will allow this? (SATA RAID, COPAN, or ???) • What happens to media? • How do we ensure preservation of these data over time? Evolution of Archive Technologies at the National Snow and Ice Data Center Presented at the THIC Conference, Boulder CO, July 19-20, 2005

More Related