1 / 32

ESDIS Status

ESDIS Status. Richard Ullman ESDIS Project Richard.E.Ullman@nasa.gov. Outline. ESDIS General Status HDF-EOS Plans Website http://hdfeos.gsfc.nasa.gov. ESDIS Science Data Services Today.

malo
Download Presentation

ESDIS Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ESDIS Status Richard Ullman ESDIS Project Richard.E.Ullman@nasa.gov HDF & HDF-EOS Workhop VI

  2. Outline • ESDIS General Status • HDF-EOS Plans • Website http://hdfeos.gsfc.nasa.gov HDF & HDF-EOS Workhop VI

  3. ESDIS Science Data Services Today • EOSDIS provides support for high data volumes from Aqua, Terra, and Landsat 7, and continues to support QuikSCAT, ACRIMSat, SAGE-III, JASON and pre-EOS-era data including TRMM, UARS, TOPEX/Poseidon, RADARSat, and others. • EOSDIS finalizing preparations to support ICESat and SORCE • In all, EOSDIS is providing data processing, archival, and/or distribution for over 15 Earth science satellite missions. • EOSDIS has set a new benchmark for data management. The total volume of the science data in our archives totals over 2 Petabytes. Since 1998, the science data volume managed by the EOSDIS has increased eight-fold, and continues to grow at a rate of over 2 Terabytes per day • In FY02, EOSDIS provided more than 16 million data and information products to over 1.8 million individuals. HDF & HDF-EOS Workhop VI

  4. 2 Petabytes Archive Growth Archive Volume has doubled each year for the past three years Science Operations Timeline Heritage Missions T/P, UARS, TOMS, ERBE Science Operations Supported Landsat 7 QuikSCAT SAGE III JASON Terra ACRIMSat Aqua GRACE SeaWiFS Radarsat TRMM 15 Million Products Distributed FY94/95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 HDF & HDF-EOS Workhop VI

  5. EOSDIS DAAC Data delivery FY’02October 1, 2002 through September 30, 2002 • Over 1.8 million distinct users • 16,013,980 products delivered • Notes to charts on following slides: • Distinct users includes users accessing DAAC web pages, including web-crawlers • Distinct user type is based on email addresses of users or URLs • Product delivered is defined as the smallest deliverable unit of data • Product delivery breakdown is based on email addresses of users receiving ECS and Non-ECS data • “FTP Delivery” are to URLs not mapped to specific domains • “Foreign Other” includes foreign email addresses whose country is known but domain-type could not be determined HDF & HDF-EOS Workhop VI

  6. 16,013,980 Data Products DeliveredOctober 1, 2001 - September 30, 2002 HDF & HDF-EOS Workhop VI

  7. Over 1.8 Million Distinct UsersOctober 1, 2001 - September 30, 2002 HDF & HDF-EOS Workhop VI

  8. ESDIS Status • After years of development, ECS is operational and generally recognized as successful. • Primary ECS development contract is essentially completed. • ECS Maintenance and Development (EMD) will emphasize maintenance more than development. • Already the majority of ESDIS budget has shifted from development to operations. • Still two major areas of new capability (see posters at AGU for more information): • Data Pools • ECHO HDF & HDF-EOS Workhop VI

  9. HDF-EOS Plans • HDF-EOS 5 development is nearly complete. • Continue to maintain, port to newer operating systems, bug-fix. • Need the advise of this community workshop - we will discuss this afternoon. • What tools or capabilities are now needed? • HDF-EOS 2 and HDF-EOS 5 • When is it the right time to press EOS science teams to migrate to HDF-EOS 5? • What steps should NASA take to facilitate? HDF & HDF-EOS Workhop VI

  10. User-defined Views, Presentations and Data Access Requests Data Providers Data Items Data Producers End Users Other Data Pools Geo PIPE Value-Added Providers Data Tailoring Workflow Management Other Data Pools Data Services Data Service Developers Dynamic Web and FTP Data Views, User-specified Data Access Data Pools Concept: The Right Data, the Right Way, Right Now! PIPE = Personalized Information Presentation Engine HDF & HDF-EOS Workhop VI

  11. General Capabilities • The Right Data: Data Location • Groups, themes, bookmarks and views • Navigation and machine-based location • Science views (e.g., science metadata) • Applications views (e.g., OGIS coverage server) • Location aids (e.g., geopolitical overlays) • External location-support services • The Right Way: Data Tailoring • Data reduction, manipulation and reformatting services • Virtual data products • Workflow management and execution monitoring • External tailoring services • Right Now: Rapid Access • Low latency data transfers • Secure remote file access • On-the-fly data compression • Automated request routing and load balancing • Near real-time data HDF & HDF-EOS Workhop VI

  12. ECHO is a metadata clearinghouse A single Internet portal for Earth science metadata search Index of data provider inventory-level data holdings metadata. .ECHO is a data order broker Forwards orders for data discovered to the data providers to fill. Data providers retain customer fulfillment service ECHO is a data service broker Registered service are associated with registered datasets Four kinds of service association Advertised, Context Passing, Brokered, Order Option ECHO is an open client API for custom user clients The EOS Data Gateway (EDG) is ESDIS’ ECHO client EOS ClearingHOuse (ECHO) http://eos.nasa.gov/echo HDF & HDF-EOS Workhop VI

  13. ECHO Data Providers • Current ECHO Data Providers • EOSDIS Core System DAACs (17% thus far) • EDC Land Processes DAAC • Goddard DAAC • NSIDC DAAC • ORNL DAAC (100%) • ECHO holds the metadata for over 3 million granules, and growing HDF & HDF-EOS Workhop VI

  14. HDF-EOS Tools and Information Web Sitehttp://hdfeos.gsfc.nasa.gov Richard.E.Ullman@nasa.gov HDF & HDF-EOS Workhop VI

  15. http://hdfeos.gsfc.nasa.gov HDF & HDF-EOS Workhop VI

  16. Website status • A resource for discovering about hdf-eos in particular. • This workshop series’ presentation archive • Links to hdf-eos tools • Site has been revamped according to comments received at the last workshop. • Tools download page now has opportunity for user feedback. • Workshop presentations are keyword searchable • New features planned • Better introductory material. • Post and organize documentation of HDF-EOS. • Better navigation to NCSA site for HDF • Host hdf-eos “web forum” • Incorporate the eostools@eos.nasa.gov listserv HDF & HDF-EOS Workhop VI

  17. HDF-EOS Profile Richard Ullman ESDIS Project Richard.E.Ullman@nasa.gov HDF & HDF-EOS Workhop VI

  18. History • September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities • Dec. 94 - ECS Engineering Support Directive to create HDF-EOS • June 1996, HDF-EOS v1.0 library released • Upgrades every 6 mo., • Current version 2.8 on HDF 4 • HDF5 support (called HDF-EOS 5) beginning November 2000 • Current version 5.1.3 on HDF5 HDF & HDF-EOS Workhop VI

  19. HDF-EOS Data Objects • Point • Swath • Profile (Swath subtype) • Grid • Zonal (HDF-EOS 5 only) HDF & HDF-EOS Workhop VI

  20. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different interface. • First released in 2000. • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. • Support same data structures • Added prefix ‘HE5_’ to HDF-EOS 2 functions. • Doesn’t preclude HDF5 functionality. • Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG HDF & HDF-EOS Workhop VI

  21. Basic File I/O Fill Values Compression Chunking/Tiling Swath Interface Grid Interface Point Interface Profile Interface Global (File), Group & Local Attributes External Data Files Subsetting Unix/Linux Support Threadsafe Version FORTRAN, C, C++ General Table Interface (proposed) HDF-EOS 5 Functionality HDF & HDF-EOS Workhop VI

  22. Top Level of HDF-EOS 5 Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL SWATH GRID POINT Global (file) Attributes The new ADDITIONAL Group has global (file) attributes The new functionality is added to the EH(utility) interface. HDF & HDF-EOS Workhop VI

  23. Swath Structure SwathName Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Data Fields Profile Fields Geolocation Fields Profile Field.n Local Attribute <FieldName>:<AttrName> Profile Field.1 Latitude Longitude Data Field.n Data Field.1 Time CoLatitude Each Data Field can have Attributes and/or Dimension Scales Shaded Objects are implemented in a fixed way. User doesn’t have direct access via the interface Dataset Group Attribute HDF & HDF-EOS Workhop VI

  24. Intended use Discrete points in time and/or location. Table of data linked to table of geographic information. 8-level Hierarchical, each level may contain indices to the level below HDF-EOS Point HDF & HDF-EOS Workhop VI

  25. Hierarchical links: Every level in a Point data set must be linked into the hierarchy. Before two levels can be linked, a link field must exist HDF-EOS Point HDF & HDF-EOS Workhop VI

  26. Intended use Across track scanning instruments. Sounding instruments Level 1: Geolocated Sensor Units Level 2: Geophysical Parameters HDF-EOS Swath HDF & HDF-EOS Workhop VI

  27. Data Fields Geolocation Fields Dimension Name: Scan Size: 16 Dimension Name: Track Size: 42 Map1 DataDimension: “Track” Geodimension: “Geotrack” Offset: 1 Increment: 2 HDF-EOS Swath “Brightness Temperature” “Time” Dimension Name: Geotrack Size: 21 “Latitude” “Longitude” HDF & HDF-EOS Workhop VI

  28. Intended use Variables mapped on uniform space-time grid scales Level 3 - Gridded single measurement parameters Level 4 - Modeled or derived from multiple measurements HDF-EOS Grid HDF & HDF-EOS Workhop VI

  29. Xdim Size: 2000 Projinfo Ydim Size: 800 Altitude Size 30 HDF-EOS Grid HDF & HDF-EOS Workhop VI

  30. Projections Supported Geographic Transverse Mercator Universal Transverse Mercator Hotine Oblique Mercator Space Oblique Mercator Polar Stereographic Lambert Azimuthal Equal Area Lambert Conformal Conic Polyconic Interrupted Goode’s Homolosine Integerized Sinusoidal Compression Methods Run-Length Encoding Adaptive Huffman Gzip HDF-EOS Grid HDF & HDF-EOS Workhop VI

  31. Level 0 - Reconstructed, unprocessed instrument/pay-load data at full resolution; any and all communications artifacts, e.g., synch. frames, communications headers, duplicate data removed. Level 1A- Reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data. Level 1B - Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent). Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data. Level 3 - Variables mapped on uniform space-time grid scales, usually with some completeness and consistency. Level 4 - Model output or results from analyses of lower level data, e.g., variables derived from multiple measurements. Product Levels HDF & HDF-EOS Workhop VI

  32. Resources • HDF-EOS on the web: • http://hdfeos.gsfc.nasa.gov/ • http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html • HDF and HDF5 on the web: • http://hdf.ncsa.uiuc.edu/ • HDF-EOS and HDF via email: • eostools@eos.nasa.gov • hdfhelp@ncsa.uiuc.edu HDF & HDF-EOS Workhop VI

More Related