Performance Characterization and Best Practices in ESX 3.5

1. AP02 NFS & iSCSI: Performance Characterization and Best Practices in ESX 3.5 Priti Mishra MTS, VMware Bing Tsai Sr. R&D Manager, VMware

2. Housekeeping • Please turn off your mobile phones, blackberries and laptops • Your feedback is valued: please fill in the session evaluation form (specific to that session) & hand it to the room monitor / the materials pickup area at registration • Each delegate to return their completed event evaluation form to the materials pickup area will be eligible for a free evaluation copy of VMware’s ESX 3i • Please leave the room between sessions, even if your next session is in the same room as you will need to be rescanned

3. Topics • General Performance Data and Comparison • Improvements in ESX 3.5 over ESX 3.0.x • Performance Best Practices • Troubleshooting Techniques • Basic methodology • Tools • Case studies

4. Key performance improvements since ESX3.0.x (1 of 3) • NFS • Accurate CPU accounting further improves load balancing among multiple VMs • Optimized buffer and heap sizes • Improvements in TSO support • TSO (TCP segmentation offload) improves large writes • H/W iSCSI (with QLogic 405x HBA) • Improvements in PAE (large memory) support • Results in better multi-VM performance in large systems • Minimized NUMA performance overhead • This overhead exists in physical systems as well • Improved CPU cost per I/O

5. Key performance improvements since ESX3.0.x (2 of 3) • S/W iSCSI (S/W-based initiator in ESX) • Improvements in CPU costs per I/O • Accurate CPU accounting further improves load balance among multiple VMs • Increased maximum transfer size • Minimizes iSCSI protocol processing cost • Reduces network overhead for large I/Os • Ability to handle more concurrent I/Os • Improved multi-VM performance

6. Key performance improvements since ESX3.0.x (3 of 3) • S/W iSCSI (continued) • Improvements in PAE (large memory) support • CPU efficiency much improved for systems with >4GB memory • Minimizing NUMA performance overhead

7. Performance Experiment Setup (1 of 3) • Workload: Iometer • Standard set based on • Request size • 1k, 4k, 8k, 16k, 32k, 64k, 72k, 128k, 256k, 512k • Access mode • 50% read/ write • Access pattern • 100% sequential • 1 worker, 16 Outstanding I/Os • Cached runs • 100MB data disks to minimize array/server disk activities • All I/Os served from server/array cache • Gives upper bound on performance

8. Performance Experiment Setup (2 of 3) • VM information • Windows 2003 Enterprise Edition • 1 VCPU; 256 MB memory • No file system used in VM (Iometer sees disk as physical drive) • No caching done in VM • Virtual disks located on RDM device configured in physical mode • Note: VMFS-formatted volumes are used in some tests where noted

9. Performance Experiment Setup (3 of 3) • ESX Server • 4-socket, 8 x 2.4GHz cores • 32GB DRAM • 2 x Gigabit NICs • One for vmkernel networking: used for NFS and software iSCSI protocols • One for general VM connectivity • Networking Configuration • Dedicated VLANs for data traffic isolated from general networking

10. How to read performance comparison charts • Throughput • Higher is better • Positive is better  higher throughput • Latency • Lower is better • Negative is better  lower response time • CPU cost • Lower is better • Negative is better  reduced CPU cost • How does this metric matter?

11. CPU Costs • Why is CPU cost data useful? • Determines how much I/O traffic the system CPUs can handle • How many I/O-intensive VMs can be consolidated in a host • How to compute CPU cost • Measure total physical CPU usage in ESX • esxtop counter: Physical Cpu(_Total) • Normalize to per I/O or per MBps • Example: MHz/MBps = {(Physical CPU usage percentage out 100%) ) X (# of physical CPUs) X (CPU MHz rating)} / (throughput in MBps)

12. Performance Data • First set: Relative to baselines in ESX 3.0.x • Second set: Comparison of storage options using Fibre Channel data as the baseline • Last: VMFS vs. RDM physical

13. Software iSCSI – Throughput Comparison to 3.0.x: higher is better

14. Software iSCSI – Latency Comparison to 3.0.x: lower is better

15. Software iSCSI – CPU Cost Comparison to 3.0.x: lower is better

16. Software iSCSI – Performance Summary • Lower CPU costs • Can lead to higher throughput for small IO sizes when CPU is pegged • CPU costs per IO also greatly improved for larger block sizes • Latency is lower • Especially for smaller data sizes • Read operations benefit most • Throughput levels • Dependent on workload • Mixed read-write patterns show most gain • Read I/Os show gains for small data sizes

17. Hardware iSCSI – Throughput Comparison to 3.0.x: higher is better

18. Hardware iSCSI – Latency Comparison to 3.0.x: lower is better

19. Hardware iSCSI – CPU Cost Comparison to 3.0.x : lower is better

20. Hardware iSCSI – Performance Summary • Lower CPU costs • Results in higher throughput levels for small IO sizes • CPU costs per IO are especially improved for larger data sizes • Latency is better • Smaller data sizes show the most gain • Mixed read-write and read I/Os benefit more • Throughput levels • Dependent on workload • Mixed read-write patterns show most gain for all block sizes • Pure read and write I/Os show gains for small block sizes

21. NFS – Performance Summary • Performance also significantly improved in ESX 3.5 • Data now shown here for interest of time

22. Protocol Comparison • Which storage option to choose? • IP Storage vs. Fibre Channel • How to read the charts? • All data is presented as ratio to the corresponding 2Gb FC (Fibre Channel) data • If the ratio is 1, the FC and IP protocol data is identical; if < 1, FC data value is larger

23. Comparison with FC: Throughput if < 1, FC data value is larger

24. Comparison with FC: Latency lower is better

25. VMFS vs. RDM • Which one has better performance? • Data shown as ratio to RDM physical

26. VMFS vs. RDM-physical: Throughput higher is better

27. VMFS vs. RDM-physical: Latency lower is better

28. VMFS vs. RDM-physical: CPU Cost lower is better

29. Topics • General Performance Data and Comparison • Improvements in ESX 3.5 over ESX 3.0.x • Performance Best Practices • Troubleshooting Techniques • Basic methodology • Tools • Case studies

30. Pre-Deployment Best Practices: Overview • Understand the performance capability of your • Storage server/array • Networking hardware and configurations • ESX host platform • Know your workloads • Establish performance baselines

31. Pre-Deployment Best Practices (1 of 4) • Storage server/array: a complex system by itself • Total spindle count • Number of spindles allocated for use • RAID level and stripe size • Storage processor specifications • Read/write cache sizes and caching policy settings • Read-Ahead, Write-Behind, etc. • Useful sources of information: • Vendor documentation: manuals, best practice guides, white papers, etc. • Third-party benchmarking reports • NFS-specific tuning information: SPEC-SFS disclosures in http://www.spec.org

32. Pre-Deployment Best Practices (2 of 4) • Networking • Routing topology and path configurations: # of links in between, etc. • Switch type, speed and capacity • NIC brand/model, speed and features • H/W iSCSI HBAs • ESX host • CPU: revision, speed and core count • Architecture basics • SMP or NUMA? • Disabling NUMA is not recommended • Bus speed, I/O subsystems, etc. • Memory configuration and size • Note: NUMA nodes may not have equal amount of memory

33. Pre-Deployment Best Practices (3 of 4) • Workload characteristics • What are the smallest, largest and most common I/O sizes? • What is the read%? write%? • Is access pattern sequential? random? mixed? • Response time more important or aggregate throughput? • Response time variance an issue or not? • Important: know the peak resource usage, not just the average

34. Pre-Deployment Best Practices (4 of 4) • Establish performance baselines by running standardized benchmarks • What’s the upperbound IOps for small I/Os? • What’s the upperbound MBps? • What’s the average/worst case response time? • What’s the CPU cost of doing I/O?

35. Additional Considerations (1 of 3) • NFS parameters • # of NFS mount points • Multiple VMs using multiple mount points may give higher aggregate throughput with slightly higher CPU cost • Export option on NFS server affects performance • iSCSI protocol parameters • Header digest processing: slight impact on performance • Data digest processing: turning off may result in • Improved CPU utilization • Slightly lower latencies • Minor throughput improvement • Actual outcome highly dependent on workload

36. Additional Considerations (2 of 3) • NUMA specific • If only one VM is doing heavy I/O, may be beneficial to pin the VM and its memory to node 0 • If CPU usage is not a concern; no pinning necessary • On each VM reboot, ESX Server will place it on the next adjacent NUMA node • Minor performance implications for certain workloads • To avoid this movement, VM should be affinitized using VI client • SMP VMs • For I/O workloads within an SMP VM that migrate frequently between VCPUs • Pin the guest thread/process to a specific VCPU • Some versions of Linux has KHz timer rate and may incur high overhead

37. Additional Considerations (3 of 3) • CPU headroom • Software initiated iSCSI and NFS protocols can consume significant amount of CPU in certain I/O patterns • Small I/O workloads require large amount of CPU; ensure that CPU saturation does not restrict I/O rate • Networking • Avoid link over-subscription • Ensure all networking parameters or even the basic gigabit connection is consistent across the full network path • Intelligent use of VLAN or zoning to minimize traffic interference

38. General Troubleshooting Tips (1 of 3) • Identify • Components in the whole I/O path • Possible issues at each layer in the path • Check all hardware & software configuration parameters, in particular • Disk configurations and cache management policies on storage server/array • Network settings and routing topology • Design experiments to isolate problems, such as: • Cached runs • Use a small file or logical device, or a physical host configured with RAM-disks: Minimizing physical disk effects • Indicate upper-bound throughput and I/O rate achievable

39. General Troubleshooting Tips (2 of 3) • Run tests with single outstanding I/O • Easier for analysis on packet traces • Throughput entirely dependent on I/O response times • Micro benchmarking each layer in the I/O path • Compare to non-virtualized, native performance results • Collect data • Guest OS data: But don’t trust the CPU% • Esxtop data • Storage server/array data: Cache hit ratio, storage processor busy%, etc. • Packet tracing with tools like TCPdump, Ethereal, Wireshark, etc.

40. General Troubleshooting Tips (3 of 3) • Analyze performance data • Do any stats, e.g., throughput or latency, change drastically over time? • Check esxtop data for anomalies, e.g., CPU spikes or excessive queueing • Server/array stats • Compare array stats with ESXstats • Is cache hit ratio reasonable? Storage processor overloaded? • Network trace analysis • Inspect packet traces to see if • NFS and iSCSI requests are processed timely • IO sizes issued by the guest match the transfer sizes over the wire • Block addresses aligned to appropriate boundaries?

41. Isolating Performance Problems: Case Study#1 (1 of 3) • Symptoms • Throughput can reach Gigabit wire speed doing 128KB sequential reads from a 20GB LUN on an iSCSI array with 2GB cache • Throughput degrades for larger data sizes beyond 128KB • From esxtop data • CPU utilization also lower for l/O sizes larger than 128KB • CPU cost per I/O is in expected range for all I/O sizes

42. Isolating Performance Problems: Case Study#1 (2 of 3) • From esxtop or benchmark output • I/O response times in the 10 to 20ms range for the problematic IOs • Indicates constant physical disk activities required to serve the reads • From network packet traces • No retransmissions or packet loss observed indicating no networking issue • Packet time stamps indicating array takes 10ms to 20ms to respond to a read request, no delay in the ESX host • From cached run results • No throughput degradation above 128KB! • Problem exists only for file sizes exceeding cache capacity • Array appears to have cache-management issues with large sequential reads

43. Isolating Performance Problems: Case Study#1 (3 of 3) • From native tests to same array • Same problem observed • From the administration GUI of the array • Read-ahead policies set to highly aggressive • Is the policy appropriate for the workload? Solution • Understand performance characteristics of the array • Experiment with different read-ahead policies • Try turning off read-ahead entirely to get the baseline behavior

44. Isolating Performance Problems: Case Study#2 (1 of 4) • Symptoms • 1KB random write throughput much lower (< 10%) than sequential writes to a 4GB vmdk file located on an NFS server • Even after extensive warm-up period • But very little difference in performance between random and sequential reads • From NFS server spec • 3GB read/write cache • Most data should be in cache after warming up

45. Isolating Performance Problems: Case Study#2 (2 of 4) • From esxtop and application/benchmark data • CPU% utilization lower but CPU cost per I/O mostly same regardless of randomness • Not likely a client side (i.e., ESX host) issue • Random write latency in the 20ms range • Sequential write < 1ms • From NFS server stats • cache hit% much lower for random writes, even after warm-up

46. Isolating Performance Problems: Case Study#2 (3 of 4) • From cached runs to a 100MB vmdk • Random write latency almost matches sequential write • Again, suggests that issue is not in ESX host • From native tests • Random and sequential write performance is almost same • From network packet traces • Server responds to random writes in 10 to 20ms, sequential writes in <1ms • Offset in NFS WRITE requests is not aligned to power-of-2 boundary • Packet traces from native runs show correct alignment

47. Isolating Performance Problems: Case Study#2 (4 of 4) • Question • Why are sequential writes not affected? • NFS Server file system idiosyncrasies • Manages cache memory at 4KB granularity • Old blocks are not updated in place; writes go to new blocks • Each < 4KB write incurs a read from the old block • Aggressive read-ahead masks the read latency associated with sequential writes Solution • Use disk alignment tool in the guest OS to align disk partition • Alternatively, use unformatted partition inside guest OS

48. Summary and Takeaways • IP-based storage performance in ESX is being constantly improved; Key enhancements in ESX 3.5: • Overall storage subsystem • Networking • Resource scheduling and management • Optimized NUMA, multi-core, and large memory support • IP-based network storage technologies are maturing • Price/performance can be excellent • Deployment and troubleshooting could be challenging • Knowledge is key: server/array, networking, host, etc. • Stay tuned for further updates from VMware

49. Questions? NFS & iSCSI – Performance Characterization and Best Practices in ESX 3.5 Priti Mishra & Bing Tsai VMware