1 / 14

Polar Constellation Special Session Data Access and Top Challenges

Polar Constellation Special Session Data Access and Top Challenges. by: Chris Sisko NESDIS/OSPO Telephone: 301-817-4783 eMail: Chris.A.Sisko@noaa.gov. Presentation Outline ESPC/NDE Operations (today) NDE product generation/distribution performance (today) Data access

zhen
Download Presentation

Polar Constellation Special Session Data Access and Top Challenges

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Polar Constellation Special Session Data Access and Top Challenges by: Chris Sisko NESDIS/OSPO Telephone: 301-817-4783 eMail: Chris.A.Sisko@noaa.gov

  2. Presentation Outline • ESPC/NDE Operations (today) • NDE product generation/distribution performance (today) • Data access • Today (Legacy DDS, Mcidas, NDE, etc) • Near Future (NDE 2.0 / PDA) • Infrastructure capacity • Challenges • Final status

  3. JPSS Mission Partners Forum

  4. NDE PE1 (Ops) System Performance – from the month of May Reporting period: 04/29/2014 - 05/28/2014

  5. Data Access (Today) http://www.ospo.noaa.gov/Organization/About/access.html • Access granted on a product by product basis with OSPO senior management approval – DAR process. • Priority access is given to organizations with: • Mission and statutory authority that supports the requirement for data access; • Signed NESDIS cooperative agreements or other legislative authorities; and • Demonstrated timeliness requirement for near-real time satellite data and products in to support operational user applications. • Requests from an organization can not exceed IT system availability or capacity limitations. • Default: PULL transfers (no double / multiple PULLs). • Only certain organizations are allowed to use PUSH transfers.

  6. Data Access (near Future) – PDA era • (New) Access granted with OSPO senior management approval : • Authorized to access to product groups • Given a data volume cap (daily limit) • Assigned priority (highest given to operational missions) • Permissions to new products are operator controlled; however, new access permissions still require OSPO management approval and International Affairs review if request is from an international requestor. • User manages their own data while OSPO operations manages the system. • PDA system is customer centric, but under extreme or peak loads (rare instances) critical users receive top priority to the system resources. • Supported protocols: FTPS, SFTP and HTTPS (pull only) • Also supported: OGC interface (NWS only: ad-hoc requests from WFOs)

  7. Examples of Data Volumes (ingress) Source: Environmental Satellite Processing and Distribution System (ESPDS) Development (OSD-ESPDS-SRD-R30-001) PDA System Requirements Document Version 5.0 March 2014

  8. External internet traffic to STAR and other authorized partners. NCEP, NWS & DOD Does not include IDPS AFWA connection OSPO/SPSD 10G example

  9. Challenges Migration over to FTPS has been difficult for some of our external users – requires coordinated firewall and host based FW changes. ESPC has experienced instances of significant data latency across internet to some of our users; this effect amplifies as more datasets are added to existing user subscriptions. NDE 1.0 external distribution capacity is 8 TB/day (today’s data volume egress is currently at 1.5 TB/day). Distribution capacity only increases once PDA becomes operational in Q4/FY2015 (PDA distribution capacity supports 65 Gbps equivalent to ~685 TB/day sustained).

  10. Challenges • Manage resources and schedule for parallel integration and testing activities at both NSOF and CBU. • Integrate, verify and validate NDE 2.0 – includes simultaneous processing of S-NPP and JPSS-1 data flows starting in 2015. • Address necessary LAN/WAN infrastructure upgrades to handle new missions producing large data volumes. Two stages of work scheduled to accomplish this: • Upgrade NOAA MAN from 1Gbps to 10Gbps (still have comms within the ESPC environment that are constrained to 1 Gbps) – underway now. • Upgrade ESPC LAN/WAN to support larger data volumes (~2016)

  11. Final Status Notes NDE 1.0 product generation and distribution system is functioning well (99.9% availability since PE-1, ops system, went operational on Sep 27, 2013). NDE test system (PE2) recently handed over to OSPO (June 20, 2013). Both systems (PE1 and PE2) have been augmented with additional Linux processing capacity and within the next month or two all AIX based algorithms will be deprecated and will run under the Linux environment.

  12. Today’s Agenda

  13. Additional Background Slides

  14. Figure 1. Estimated bandwidth necessary to transfer one full orbit of SNPP data as a function of time. SNPP generates a data volume of approximately 4 TB/day (uncompressed) - equates to approximately 290 GB/orbit . The depicted bandwidth curve is rough estimate (does not include network overhead or contention).

More Related