DADDS DCS LRIT Prototype LRIT Latency Results

1. DADDS DCS LRIT PrototypeLRIT Latency Results Presented by Microcom Design, Inc. May 2014

2. Task Description • Authorized under a DCS Sustaining Engineering Contract. • Develop a DADDS Process to create and disseminate LRIT DCS Message files. • Transfer files to LRIT system via SFTP. • Develop text based and graphical tools to analyze latency. • Utilize Microcom LRIT Receiver to monitor data flow and determine end-to-end latency. • DAMS-NT to DADDS to LRIT to Reception • Include configurable settings to determine file generation: • Message Count • Byte Count • Time – timeout to minimize latency Microcom Design, Inc.

3. DOMAST versus LRIT Comparison • DOMSAT streams DCS messages, while LRIT transfers files. • For efficiency, multiple DCS messages must be collected into files. • DOMSAT stream “dedicated” to DCS, LRIT stream shares multiple NOAA products; DCS, EMWIN, Imagery. • DOMSAT uses Domestic Satellite, LRIT broadcast over GOES. • DOMSAT requires annual funding, LRIT has no user recurring satellite usage costs. • DOMSAT has limited coverage outside of CONUS, LRIT has hemispherical coverage. • DOSAT primarily used for other data, GOES/LRIT dedicated to environmental usage. • DOMSAT frequency in Ku band, while LRIT in L band. • DOMSAT highly susceptible to weather fading, LRIT far more robust. Microcom Design, Inc.

4. LRIT Operational Summary 1 • Downlink Frequency: 1691 MHz (L Band) • Information Data Rate: 128 kbps (16 kBps) • Forward Error Correction: Convolution and Block • Inner Code: Rate ½ Viterbi, Constraint Length 7 • Outer Code: Reed Solomon (255,223) – 223 information bytes and 32 check bytes; can correct up to 16 bytes in error • Data Files are Packetized and Framed • Frame: 1024 Bytes • 4-byte FSS, followed by four interleaved 255 byte RS blocks. • ~ 880 bytes of information in each frame. • Packets: Variable up to 8 kilobytes • Each Packet terminated in CRC-16. • Large Files require multiple packets. • First Packet contains LRIT Headers. Microcom Design, Inc.

5. LRIT Operational Summary 2 • Files can be sent on one of 63 Virtual Channels (VC) • Each Frame begins with a 6-byte sequence that defines the VC and includes a 3-byte Frame Counter. • Frame Sequence Counters are unique to VC. • Virtual Channels Represent a Priority • The lower the VC number, the higher the priority. • Priority scheme allows smaller, high priority files to be interspersed with larger, low priority files. • EMWIN uses VC 00; highest priority. • LRIT is a Continuous Transmission • A 64TH Virtual Channel (VC 63) is used for “Fill”, i.e. when there is no data to send. • Actual transmission rate is 292.7 ksps (kbps) 128 = (292.7 / 2) * (223/255) Microcom Design, Inc.

6. LRIT DCS Initial Analysis – Frame Utilization 1 Microcom Design, Inc.

7. LRIT Initial Analysis – Frame Utilization 2 • Initial analysis showed that the LRIT stream was dominated by long periods of either DCS, Imagery, or Fill, but … • EMWIN maintained steady flow of data. Microcom Design, Inc.

8. LRIT Initial Analysis – Product Utilization • Initial Frame Analysis Showed: • EMWIN: ~ 7.3 kbps ~5.7% • DCS: ~15.3 kbps ~12.0% • Imagery: ~41.2 kbps ~32.2% • Fill: ~64.2 kbps ~50.1% • Initially: • EMWIN used VC00 • DCS used VC18 (High), VC38 (Medium), VC53 (Low) • Imagery used VCs 13, 23, 33, 41, and 43 • More Recently: • EMWIN continues to use VC00 • DCS was using 1, 48, and 53 • Imagery shared VC53 with Low Priority DCS Files Microcom Design, Inc.

9. LRIT DCS File Summary • High Priority Files: • Contain one of two platforms (or both) that transmit on six minute intervals at the top of the minute. • Being used by NOS and NWS to facilitate their latency testing. • Files sent in 1 Packet/Frame. • Medium Priority Files: • Consist of approximately 500 platforms. • All platforms appear to transmit on DCS channels 40-49. • Files typically require 1 Packet sent over several Frames. • Low Priority Files: • Everything not in Low or Medium priority files. • Files are quite large (~200 kilobytes, ~1000 messages). • Files typically sent about every 2 minutes. Microcom Design, Inc.

10. LRIT Baseline Latency • Typical composite latency ~100 seconds with regular peaks over 300 seconds; some as high as 600 seconds. • Peaks believed to be primarily due to DCS Low Priority files sharing same VC as imagery files. Microcom Design, Inc.

11. Prototype Target Goals • Create compatible DCS LRIT files from DADDS. • Determine if the LRIT system can handle smaller more frequent files. • Send all DCS messages as High priority. • Accurately measure overall LRIT DCS latency. • Breakdown latency into constituent components. • See if DADDS can provide improvement over existing dissemination path. Microcom Design, Inc.

12. Initial Live Test Run • Test was run on April 16, 2014 from approximately 14:24:30 to 14:29:30 UTC. • All data sent as High Priority. • DCS data collected and sourced from NSOF DADDS. • Messages collected into files of approximately 8 kilobytes each. • Total of 58 files generated and transmitted. • Approximately 2450 DCS messages were transmitted and received. • Test run was performed while WCDA DCS LRIT was also flowing. • No anomalies were detected and test ran as expected. • After confirming data was flowing and no issues were detected, performance data was collected from approximately 14:27:00 to 14:29:30 UTC. Microcom Design, Inc.

13. 1ST Test – Frame Utilization Graph Microcom Design, Inc.

14. 1ST Test – Frame Utilization Notes • LRIT Frames counted for DADDS DCS, EMWIN, Fill, and Other. • Other includes both Imagery and the DCS Data coming from WCDA. • A data sample set occurs every 100 LRIT Frames. • NSOF DADDS DCS Data accounted for approximately 10% of the Total Frames. • Average: 9.8 Frames per 100 • Minimum: 0 • Maximum: 28 • EMWIN accounted for approximately 6% of the Total Frames. • Insertion of a second DCS stream did not create any major issues. • Only impact was a slight reduction in the transmission rate of the “Other” frames. • This short duration test showed minimal Fill packets (~7%). • Longer term testing has shown that the LRIT typically only runs at 65-70% utilization (30-35% Fill). Microcom Design, Inc.

15. 1ST Test – Overall Results • Data points generated and accumulated every 100 messages. • Computer generating graph had clock running 5-6 seconds fast. Microcom Design, Inc.

16. 1ST Test – Overall Notes • Latency calculated from the time the message was received to time it was ingested by test computer at Microcom. • Average DCS message latency for this test run calculated on every 100 messages. • Note: After the test run was complete, the test computer’s clock was checked and was found to be 5-6 seconds fast. • Accounting for computer’s clock error, average latency for the initial test run was actually on the order of 45-50 seconds. Microcom Design, Inc.

17. 1ST Test – LRIT File Latency Histogram Microcom Design, Inc.

18. 1ST Test – LRIT Latency Notes • Latency calculated in seconds from file generation time to reception by LRIT Receiver. • Average: 25 seconds • Minimum: 19 seconds • Maximum: 35 seconds • This latency encompasses the transfer time from DADDS-to-LRIT, the LRIT processing time, and the transfer time from the LRIT processor to the LRIT uplink transmitter. • Note: The times are approximately 3-5 seconds higher than actual. • Microcom utilized the first DCS message in the file to generate the filename timestamp. • A better indication would be provided by using the time the last message was added to the file. • Average file latency is predicted to be more on the order of 20 seconds or so. • Average Latency remained relatively consistent over initial run. Microcom Design, Inc.

19. 1ST Test – DADDS LRIT File Generation Microcom Design, Inc.

20. 1ST Test – DADDS LRIT Latency Notes • Latency calculated in seconds from end time of first message in file to file timestamp: • Average: 16.4 seconds • Minimum: 13.5 seconds • Maximum: 18.6 seconds • DADDS file latency on the order of 16 seconds. • Higher than expected, but optimizing DADDS process was not a primary goal of this task. • Microcom believes DADDS latency can be easily reduced to 5-10 seconds. • Latency was consistent over initial run. Microcom Design, Inc.

21. Second Live Test Run • Test was run on April 23, 2014 from approximately 13:27:30 to 16:05:00 UTC. • All data sent as High Priority. • DCS data collected and sourced from NSOF DADDS. • Messages collected into files of approximately 8 kilobytes each. • A total of 2,035 files were generated, transmitted, and received. • Over 76,000 DCS messages were transmitted and received (~29,000 messages per hour). • Test run was performed without WCDA DCS LRIT data flowing. • No operational issues were detected; independent confirmation of reception from NOS and NWS. • Performance data was collected from almost the entire run UTC. Microcom Design, Inc.

22. 2ND Test – Frame Utilization Graph Microcom Design, Inc.

23. 2ND Test – Frame Utilization Notes • LRIT Frames counted for DCS, EMWIN, Fill, and Other; Other primarily includes Imagery. • A data sample set occurs every 100 LRIT Frames. • DCS Data accounted for 10.24% of the Total Frames. • Average: 10.24 Frames per 100 • Minimum: 0 • Maximum: 45 • Stnd Dev: 7.6 • EMWIN accounted for 5.73% of the Total Frames. • Other (Imagery) accounted for 57.14% of the Total Frames. • Fill accounted for 26.89% of the Total Frames. • EMWIN and DCS packets flowed at relatively consistent rate, while Imagery & Fill dominated alternately. Microcom Design, Inc.

24. 2ND Test – Frame Utilization Hour 14 Microcom Design, Inc.

25. 2ND Test – Overall Results • Data points generated and accumulated every 100 messages. Microcom Design, Inc.

26. 2ND Test – Overall Notes • Latency calculated from the time the message was received to time it was ingested by test computer at Microcom. • Average end-to-end latency for the bulk of this run was on the order of 40 seconds. • From ~15:05 to ~15:25 the average latency gradually increased to a peak of over 65 seconds and then came back down to ~40 seconds. • Standard Deviation of Latency consistently remained under 5 seconds. Microcom Design, Inc.

27. 2ND Test – Existing Dissemination Comparison • Typical composite latency 60-80 seconds with peaks over 200 seconds. • Improvement from initial baseline due to move to using VC 01 for all DCS files. Microcom Design, Inc.

28. 2ND Test – LRIT File Latency Histogram Microcom Design, Inc.

29. 2ND Test – LRIT Latency Notes • Latency calculated in seconds from file generation time to reception by LRIT Receiver. • Average: 22.9 seconds • Minimum: 13 seconds • Maximum: 42 seconds • Stnd Dev: 4.2 seconds • Latency encompassed the transfer time from DADDS-to-LRIT, the LRIT processing time, and the transfer time from the LRIT processor to the LRIT uplink transmitter. • Note: The times are approximate 3-5 seconds higher than actual. • Microcom utilized the first DCS message in the file to generate the filename timestamp. • A better indication would be provided by using the time the last message was added to the file. • Average file latency is predicted to be more on the order of 20 seconds or so. • Average Latency remained relatively consistent over entire run. Microcom Design, Inc.

30. 2ND Test – LRIT File Latency Over Run Microcom Design, Inc.

31. 2ND Test – DADDS File Latency Microcom Design, Inc.

32. 2ND Test – DADDS File Latency Notes • Latency calculated from end timestamp of first DCS message in file to the file generation time. • Average: 18.9 seconds • Minimum: 13 seconds • Maximum: 42 seconds • Stnd Dev: 6.4 seconds • Baseline average appears to be on the order of 16-17 seconds. • Discounting three spiked up sections, latency remained relatively consistent over entire run. • Excessive rise in processing delay accounts for increased overall latency shown in graph, i.e the point centered about 15:15 UTC where the overall latency increased to over 65 seconds. • Smaller spike at 14:10 in above graph also correlates to similar spike in overall graph. • At this time Microcom has no specific explanation for the increases in DADDS processing time. • Most likely cause is server performance due to some other function that required high CPU resources. • Additional investigation will be required to determine exact cause. • Should be readily addressable. Microcom Design, Inc.

33. Target Goals Acheived • Testing was highly successful and achieved target goals: • DADDS can flow DCS data to LRIT system. • DADDS DCS data files were properly formatted: • Message data was correctly received by Microcom’s LRIT Receiver, which also receives WCDA sourced DCS data. • Independent confirmation provided by NOS (Mark Bushnell, Warren Krug) and NWS (Brian Jackson) during and post test run. • LRIT system has no issues in processing smaller more frequent files. • LRIT throughput not significantly affected by sending all DCS messages as High Priority. Microcom Design, Inc.

34. Testing Conclusions • Testing provided solid baseline information on overall latency and distribution of latency in various systems. • Overall typical latency is on the order of 35-45 seconds. • Latency remains consistent over extended periods of time. • LRIT processing and distribution latency on the order of 20-25 seconds. • Typical DADDS file generation latency is on the order of 15-18 seconds. • The DADDS-to-LRIT prototype and subsequent analysis by Microcom has clearly demonstrated capability to accurately measure and to pinpoint problem areas in the overall latency. Microcom Design, Inc.

35. Possible Next Steps • The unusual and unexpected areas where latency above baseline will require additional investigation to determine the cause. However, Microcom is confident this can be mitigated. • Microcom is also confident that the DADDS file generation can be optimized and reduce the typical latency down closer to 10 seconds (or better). • Assuming an LRIT system latency of 20-25 seconds and a 5-10 second DADDS latency can be achieved, than it should be possible to achieve an average latency of 30 seconds. • Make suggested improvements and perform additional testing. Microcom Design, Inc.

36. END OF PRESENTATION “THANK YOU” FOR YOUR ATTENTION