Profiling Network Performance in Multi-tier Datacenter Applications

1. Profiling Network Performancein Multi-tier Datacenter Applications Scalable Net-App Profiler Minlan Yu Princeton University • Joint work with Albert Greenberg, Dave Maltz, Jennifer Rexford, Lihua Yuan, SrikanthKandula, Changhoon Kim

2. Applications inside Data Centers …. …. …. …. Aggregator Workers Front end Server

3. Challenges of Datacenter Diagnosis • Large complex applications • Hundreds of application components • Tens of thousands of servers • New performance problems • Update code to add features or fix bugs • Change components while app is still in operation • Old performance problems(Human factors) • Developers may not understand network well • Nagle’s algorithm, delayed ACK, etc.

4. Diagnosis in Today’s Data Center Packet trace: Filter out trace for long delay req. App logs: #Reqs/sec Response time 1% req. >200ms delay Host App Too expensive Application-specific Packet sniffer OS SNAP: Diagnose net-app interactions Switch logs: #bytes/pkts per minute Too coarse-grained Generic, fine-grained, and lightweight

5. SNAP: A Scalable Net-App Profilerthat runs everywhere, all the time

6. SNAP Architecture At each host for every connection Collect data

7. Collect Data in TCP Stack • TCP understands net-app interactions • Flow control: How much data apps want to read/write • Congestion control: Network delay and congestion • Collect TCP-level statistics • Defined by RFC 4898 • Already exists in today’s Linux and Windows OSes

8. TCP-level Statistics • Cumulative counters • Packet loss: #FastRetrans, #Timeout • RTT estimation: #SampleRTT, #SumRTT • Receiver: RwinLimitTime • Calculate the difference between two polls • Instantaneous snapshots • #Bytes in the send buffer • Congestion window size, receiver window size • Representative snapshots based on Poisson sampling

9. SNAP Architecture At each host for every connection Collect data Performance Classifier

10. Life of Data Transfer Sender App • Application generates the data • Copy data to send buffer • TCP sends data to the network • Receiver receives the data and ACK Send Buffer Network Receiver

11. Taxonomy of Network Performance Sender App • No network problem • Send buffer not large enough • Fast retransmission • Timeout • Not reading fast enough (CPU, disk, etc.) • Not ACKing fast enough (Delayed ACK) Send Buffer Network Receiver

12. Identifying Performance Problems Sender App • Not any other problems • #bytes in send buffer • #Fast retransmission • #Timeout • RwinLimitTime • Delayed ACK diff(SumRTT) > diff(SampleRTT)*MaxQueuingDelay Send Buffer Sampling Network Direct measure Receiver Inference

13. SNAP Architecture Management System Topology, routing Conn  proc/app At each host for every connection Cross-connection correlation Collect data Performance Classifier Offending app, host, link, or switch

14. Pinpoint Problems via Correlation • Correlation over shared switch/link/host • Packet loss for all the connections going through one switch/host • Pinpoint the problematic switch

15. Pinpoint Problems via Correlation • Correlation over application • Same application has problem on all machines • Report aggregated application behavior

16. SNAP Architecture Offline, cross-conn diagnosis Online, lightweight processing & diagnosis Management System Topology, routing Conn  proc/app At each host for every connection Cross-connection correlation Collect data Performance Classifier Offending app, host, link, or switch

17. Reducing SNAP Overhead • SNAP overhead • Data volume: 120 Bytes per connection per poll • CPU overhead: • 5% for polling 1K connections with 500 ms interval • Increases with #connections and polling freq. • Solution: Adaptive tuning of polling frequency • Reduce polling frequency to stay within a target CPU • Devote more polling to more problematic connections

18. SNAP in the Real World

19. Key Diagnosis Steps • Identify performance problems • Correlate across connections • Identify applications with severe problems • Expose simple, useful information to developers • Filter important statistics and classification results • Identify root cause and propose solutions • Work with operators and developers • Tune TCP stack or change application code

20. SNAP Deployment • Deployed in a production data center • 8K machines, 700 applications • Ran SNAP for a week, collected terabytes of data • Diagnosis results • Identified 15 major performance problems • 21% applications have network performance problems

21. Characterizing Perf. Limitations #Apps that are limited for > 50% of the time Send Buffer • Send buffer not large enough 1 App • Fast retransmission • Timeout Network 6 Apps • Not reading fast enough (CPU, disk, etc.) • Not ACKing fast enough (Delayed ACK) 8Apps Receiver 144 Apps

22. Three Example Problems • Delayed ACK affects delay sensitive apps • Congestion window allows sudden burst • Significant timeouts for low-rate flows

23. Problem 1: Delayed ACK • Delayed ACK affected many delay sensitive apps • even #pktsper record  1,000 records/sec odd#pktsper record  5 records/sec • Delayed ACK was used to reduce bandwidth usage and server interrupts B A Data ACK every other packet ACK Proposed solutions: Delayed ACK should be disabled in data centers …. Data 200 ms ACK

24. Send Buffer and Delayed ACK • SNAP diagnosis: Delayed ACK and zero-copy send Application buffer Application With Socket Send Buffer 1. Send complete Socket send buffer Receiver Network Stack 2. ACK Application buffer Application Zero-copy send Receiver 2. Send complete Network Stack 1. ACK

25. Problem 2: Congestion Window Allows Sudden Bursts • Increase congestion window to reduce delay • To send 64 KB data with 1 RTT • Developers intentionally keep congestion window large • Disable slow start restart in TCP Drops after an idle time Window t

26. Slow Start Restart • SNAP diagnosis • Significant packet loss • Congestion window is too large after an idle period • Proposed solutions • Change apps to send less data during congestion • New transport protocols that consider both congestion and delay

27. Problem 3: Timeouts for Low-rate Flows • SNAP diagnosis • More fast retrans. for high-rate flows (1-10MB/s) • More timeouts with low-rate flows (10-100KB/s) • Proposed solutions • Reduce timeout time in TCP stack • New ways to handle packet loss for small flows

28. Conclusion • A simple, efficient way to profile data centers • Passivelymeasure real-time network stack information • Systematically identify problematic stages • Correlate problems across connections • Deploying SNAP in production data center • Diagnose net-app interactions • A quick way to identify them when problems happen • Future work • Extend SNAP to diagnose wide-area transfers

29. Thanks!