Management of AMSR-E and GLAS Data by NSIDC

1. Management of AMSR-E and GLAS Data by NSIDC Melinda Marquis AMSR/AMSR-E Data Management Team Lead GLAS/ICESat Data Management Team Lead National Snow and Ice Data Center PoDAG Meeting April 25 & 26, 2001

2. Outline • Data Flows • Data Distribution Services • Documentation • Post-launch communications • Validation • Operations • Testing • Progress & Milestones • Subsetting • Products tables

3. AMSR-E Data Flow L0 EDOS L0

4. GLAS Data Flow

5. Distribution of data by ECS Data Management Subsystem Data Server Subsystem Science Data Server Database EOS Data Gateway (EDG) Search and Order Data Archive (Powderhorn) Granules Distributed User

6. Data Distribution Services • Data centers provide different levels of service for different products • NSIDC distributes data at the same granularity as it is archived • We plan to limit distribution of level 0 AMSR-E and GLAS data • We intend to offer additional services, such as subsetting

7. Data Support Services Provided by NSIDC • DIFs (submitted to the Global Change Master Directory) • Guide Documents (User guides) • Web Site • Newsletter • DAAC Yearbook • Outreach

8. Directory Interchange Formats (DIFs) • A DIF is a metadata document • Submitted to the NASA’s Global Change Master Directory (GCMD) http://gcmd.gsfc.nasa.gov/ • GCMD is a comprehensive directory of descriptions of data sets relevant to global change research. • A DIF is a collection of fields that allow user to decide whether a particular data set would be useful to them. • Compatible with U.S. federally mandated Federal Geographic Data Committee’s (FCDC) Content Standard on Digital Geospatial Metadata.

9. Guide Documents • Is a technical document (user’s guide) for data products funded by NASA. • Must follow a specific template • Template at (http://harp.gsfc.nasa.gov/v0ims/DOCUMENTATION/GUIDE-VALID/GUIDES/TemplateECS2.htm). • Contains detailed information about the data product, allowing an uninitiated user to understand and use the data fully and accurately.

10. Post-Launch Communications between SIPS/SCF/Science Team and NSIDC • Communication will continue to be important after launch • Either document protocol in Operations Agreement (OA) between NSIDC and SIPS or in Memorandum of Understanding (MOU) between NSIDC and SCF

11. Post-Launch Communications between SIPS/SCF/Science Team and NSIDC • NSIDC’s web site will provide information on: • Availability of products • Status of Aqua and of AMSR-E and of ICESat and of GLAS • Known errors • Any changes to processing will need to be conveyed so our documentation can be updated • So we can inform users, including registered users

12. Post-Launch Communications between SIPS/SCF/Science Team and NSIDC USER Documentation NSIDC USO Documentation Specialist NSIDC IN-HOUSE RESOURCES Science Operations Support Staff System Engineering/ Admin. Programmers Ops Science Staff As needed OUTSIDE RESOURCES Science Team SIPS Others

13. Validation – AMSR-E • Richard Armstrong, Jim Maslanik and Julienne Stroeve are actively involved in validation efforts in support of the AMSR-E science team • Therefore, they will perform very high level QA of AMSR-E data products

14. Archive Migration • Old drives and media will no longer be supported by Storage Tek. • Installation of new drives and media began in April 2001. • All data, including ASMR-E and GLAS will be archived on the new media, which will be the 9940 tapes from Storage Tek.

15. PDS (Product Distribution System) • Installation completed by the second week in April 2001. • The PDS system capabilities: • CD-R • 8MM tape • DV • Automated QC of all media (not data) • Automated label printing

16. Mission Operations and Science Systems (MOSS) Tests – Aqua/AMSR-E • Formal test exercise to demonstrate Aqua Science Operations Concurrent Aqua test load with nominal Terra operations

17. Mission Operations and Science Systems (MOSS) Tests Aqua science systems end-to-end test from EDOS through science data processing facilities (AMSR-E SIPS) to assess readiness for: • Demonstration of system performance and concurrent operations ( ingest, archive, processing, user access, and distribution) for Terra, Landsat-7 and Aqua • Verify compatibility of AMSR data products with ECS data types • Ability to support early post-launch science operations with DAAC/SIPS production using simulated data • Demonstrate subscriptions, user pull and distribution (ftp and tape) • Demonstrate nominal operations personnel and procedures • Identify critical liens, problems, or defects that must be resolved prior to Aqua launch

18. MOSS 1Summary Totals • Data Provider Data Type Granules Ingested • EDOS AE_PMSCI 21 • PM1GBAD1 19 • PM1GBAD4 2 • PM1GDAB8 2 • GHCC AE_L2A 47 • AE_Ocean 48 • AE_Land 48 • AE_Rain 48 • AE_RnGd • AE_DyOcn • AE_WkOcn • AE_M0Ocn • AE_DySno 2 • AE_5DSno 2 • AE_MoSno 2 • AE_S16 2 • AE_SI12 2 • AE_S125 2 • AE_Land3 2 • Browse 46 • PO.DAAC AMSREL1A 53

19. MOSS 2Summary Totals • Data Provider Data Type Granules Ingested • EDOS AE_PMSCI 32 • PM1GBAD1 33 • PM1GBAD4 6 • PM1GDAB8 6 • GHCC AE_L2A 56 • AE_Ocean 56 • AE_Land 56 • AE_Rain 56 • PO.DAAC AMSREL1A 56

20. Future MOSS Tests (Aqua/AMSR-E) • MOSS 3 : May 14 - May 18 • MOSS 4 : July 16 - July 20 • MOSS 5 : September 17 - September 21

21. GLAS Test Schedule • April 24: ICT -- went well. We received PDRs and PDSs for 17 L0 APIDs. All successfully ingested. • May 14-16: MOSS 3 (part of Aqua MOSS) • July 16-18: MOSS 4 (part of Aqua MOSS) • July 2-31: IGST-4 • July 10: IGST-6 • Sept. 3- Oct. 5: IGST-7

22. Test Schedule: AMSR-E and GLAS

23. Progress – AMSR-E • ECS System Upgrades needed for AMSR-E • ECS 5B and 55 Integration and Test • Install and check out Release 5B (this was completed by July 2000, i.e., before FY01) • Install and check out Release 55 Release 55 has been split into 2 parts: • a) 5B06 (Dec. 2000) • b) IRIX upgrade/SGI (2nd quarter FY01) • 5B07 (March 12 in TS1; March 19 in OPS) • System Tests • Perform Regressions Tests on Release 55 (2nd quarter FY01) • ECS 55 Site Readiness Acceptance (2nd quarter FY01) • System Confidence Tests (SCTs) (One already during 1st quarter; maybe one or more in future.) • MOSS (Mission Operations and Science Systems) Tests (First MOSS was in Dec., 2000. Second was in Feb., 2001, i.e., 2nd quarter FY01) • Interface Documents • OAs and ICDs are completed, except for those with NASDA.

24. Milestones – AMSR-E • 1st Quarter (December 31, 00) • 1.Installation and Checkout 55: • a. 5B06 (Dec. 2000) • 2.System Confidence Test • 3.MOSS Test (Dec. 2000) • 2nd Quarter (March 31, 01) • 4. IRIX upgraded • 5.Second MOSS Test (Feb.) • 3rd Quarter (June 30, 01) • 6.DIFs completed • 7.Add additional Hardware (getting PDS, another rimage system, RAID) • 4th Quarter (September 30, 01) • 8.Guide documents (summaries) completed • 1st Quarter (December 31, 01) • 9. Operations Readiness Review (Oct. 15) • 10.Launch (NET 12/01) • Level 0 ingest begins

25. Milestones -- GLAS • 1st Quarter (by December 31) • 1.Write DIFs (drafts)  • 2nd Quarter (by March 31) • 2.Polish/finish (have reviewed) DIFs • 3rd Quarter (by June 31) • 3. ICD b/w ECS at NSIDC DAAC and ISIPS • 4.OA b/w NSIDC DAAC and ISIPS • 5.OA b/w NSIDC DAAC and EDOS • 6.OA or MOU b/w NSIDC DAAC and SCF ? • 7.Interface Confidence Tests

26. Milestones – GLAS (cont.) • 4th Quarter (by September 30) • 9. Guide doc – L1 Alt. (GLA01, 05 & 06) • 10. Guide doc – L2 Alt. (GLA12-15) • 11.      Upgrades to ECS for support of ICESat • 12.Installation and Checkout • 13.Acceptance and Regression Tests • 14.Installation and Checkout • 15.Regression Tests • 16.Interface Tests • 17.Mission End-to-End Tests (IGSTs) • Launch (Dec. 15, 01) • FY02, 1st Quarter • Guide doc – L1 &L2 Atmos. (GLA02, 7-11) • Guide doc – GLA16

27. Subsetting • ESDIS is working to provide subsetting for data managed in the ECS • MODIS • AMSR-E • GLAS • For data in HDF-EOS format, the University of Alabama’s HEW subsetter may be used • GLAS data products will be in binary format, with exceptions (higher level products) • NSIDC is pursuing becoming an rSCF • NSIDC hopes to develop subsetter tool unique to GLAS

28. Backup Slides

29. AMSR-E Products

30. Products

31. Products

32. Products

33. GLAS Data Products and Granularity

34. GLAS Data Granularity and Potential User Issues • User A: • GLA12 is 49MB • 49MB/day * 183 days = 8,967 MB • Greenland Ice Sheet occupies 15% of GLA12 • 0.15*8967MB = 1345MB •  User B: • GLA16 is 117MB • 117MB/day * 183 days = 21,411 MB •  Greenland Ice Sheet occupies 5% of GLA16 • 0.05*21411MB = 1071MB

35. GLAS Data Granularity and Data Delivery Issues • A CD-ROM holds approx. 650 MB of data, an 8mm Tape holds approx. 5 GB of data. • Our typical GLA12 user would require 2 x 8mm tapes or 15 CD-ROMs. • Our GLA16 user would require 5 x 8mm tapes or 36 CD-ROMs. •  If the data were staged to an ftp server, then the user would be required to download large amounts of data, which would require large periods of time. • @56 kb/sec, the GLA12 user would take ~44.5 hrs • @56 kb/sec, the GLA16 user would take ~106.2 hrs

36. GLAS Data Granularity and Data Delivery Issues • However, if the data were subsetted for the Greenland Ice Sheet alone, then the GLA12 user would require • 1x8mm tape, 2 CD-ROMs or ~6.6 hrs of @download time • similarly the GLA16 user would require 1x8mm tape, 2 CD-ROMs or ~5.3 hrs of download time. •  This reduction in media use will reduce media distribution costs and can encourage use of ftp to deliver the data to the user. Subsetting becomes essential for efficient data distribution and is being addressed.

37. Spatial Query Server Spatial search via NOSE ECS Science Data Server (SDSRV) EDG Query Parameters AMSR-E Granule Inventory Granule List Spatial Indicies

38. AMSR-E pre-defined orbit data Implementation of Nominal Orbital Spatial Extent (NOSE) for AMSR-E ECS Science Data Server (SDSRV) AMSR-E Granule Inventory AMSR-E DLL Spatial Query Server AMSR-E NOSE Table

39. Insertion of AMSR-E granule into ECS ECS Science Data Server (SDSRV) AMSR-E Granule Inventory AMSR-E DLL AMSR-E granule metadata Spatial Query Server Spatial Indicies

40. Subscriptions 5. Email and/or Granule push to user NSIDC User Services Communications Subsystem (CSS) 4. Event is recognized as “belonging” to this user 1. USO enters subscription for user through GUI 3. DSS tells subscription Server about each event Data Server Subsystem (DSS) 2. Event occurs Ingest of data

41. QA at NSIDC • Automatic QA: • If can’t ingest -> no good • We plan to exercise media: pull data off randomly to inspect quality of media – still readable (new PDS does this, too) • Manual QA: • Inspect media when we distribute data on media (not via ftp) • We will support QA updates if provider wishes to utilize that functionality (hope to make it automated)

42. AMSR-E L1A Data -- availability Proposal: • One orbit of data will be available to team members only, three weeks after the AMSR-E instrument is turned on. • One (1) day's worth of data will be available to team members only, four weeks after the AMSR-E instrument is turned on. • All Level 1A data will be continuously available to team members after 2 months.