Benchmarking A Networked Backup System Using Natural Workloads

1. Benchmarking A Networked Backup System Using Natural Workloads Converting Interesting Data Into Useful Information Bryan Drake Virginia Information Technologies Agency

2. Overview • About VITA • Intent of this presentation • Benchmarked environment • Benchmark process • Benchmark data • Data reduction and presentation • Application of benchmark for runtime prediction • Optimal solution selection • Conclusions • Recommendations

3. About VITA • Virginia Information Technologies Agency • Created July 2003 • New central agency for computing services in Virginia • Central point for procurement and management of IT services and goods • Centralized and decentralized systems • Mainframes to desktop

4. My Intentions • Demonstrate simple benchmarking process • Data reduction and presentation converts data into information • Encourage YOU to perform benchmarks on your systems • Not intended as Rules Of Thumb - ROT • Disclaimers

5. Motivations for Benchmark • Rapid growth • Limited backup windows • Some clients have limited windows • Workload peaks on weekends and after midnight • Limited budget • Can’t afford lots of excess capacity • Service level expectations • Diverse clients • Windows and UNIX clients • General servers and database servers • Data volumes from 5 GB to 3 TB

6. Benchmark Environment

7. Benchmark Environment • Windows and UNIX servers as clients • Veritas NetBackup software • DLT 7000 tape drives • Windows master and media servers • UNIX media server clients on Sun E10000 • Benchmark information in published paper • Not presented here • Shared tape drives on fibre network

8. Theory vs. Reality • What appears to be a simple equation is actually complex • How much do you have to backup • How many hours in your backup window • How fast are the tape drives • Simple Division to find required # of Drives • Right? • Not necessarily!

9. Theory vs. Reality • Example: • 401,722,416 KB • 8 Hour Window • 7200 KBS Tape Data Rate DLT Drives • Division to derive # drives • No problem, right? • This assumes that tape drives are only bottleneck • Need to understand where bottlenecks exist and impact

10. Benchmark Process • Select representative clients • Focus on problematic clients • Large data volumes, small windows • Repetitive, strategic benchmarks • Change tuning parameters and Rerun • Develop simple predictive model • Predict run times based on various tuning parameter combinations

11. Tuning Options • Multiplexing • Combining multiple streams on a single tape • Intended to maximize tape throughput • Avoid tape stop-start processing • Multi-streaming • Sending multiple data streams from a client • Intended to maximize throughput from the client

12. Benchmark Processcontinued • Run the sample streams • Collect data • Identify steady state • Change parameters • Rerun sample streams • Reduce the data • Evaluate possible bottlenecks • Model effect of parameter changes

13. First Set of Benchmarks • Large UNIX client with fast backplane • Single 100Mbs Ethernet link • Multiplexing 1-4 • Single tape • Maximize tape throughput • Measure KBS and total KB • Look for steady state

14. Single Stream (MPX=1)Total KB

15. Single Stream (MPX=1)KB per Second (KBS)

16. MPX=2Total KB

17. MPX=2KBS

18. MPX=3Total KB

19. MPX=3KBS

20. MPX=4Total KB

21. MPX=4KBS

22. Interesting Data • Observations • Total KB increases over time DUH! • After initial fluctuations, KBS reaches steady state • Issues • Graphs not on same scale • Too much data, not enough information • Does provide a starting point • Don’t stop here!

23. Data Reduction • Use KBS as most useful metric for prediction • Use “final” KBS values • Use average of multiple Streams • Calculate tape and link throughput from stream throughput • Graph KBS vs. multiplexing value

24. Reduced & Summarized Data

25. Figure 3 Multiplexed Throughput - Single UNIX Client 9000 8000 7000 6000 5000 Stream KBS Drive/Link 4000 3000 2000 1000 0 1 2 3 4 Multiplexing Effect of Multiplexing on Stream and Tape Throughput

26. Figure 3 Multiplexed Throughput - Single UNIX Client 9000 8000 7000 6000 KBS 5000 Stream Drive/Link 4000 3000 2000 1000 0 1 2 3 4 Multiplexing Observations • More useful information • Increasing multiplexing increases tape throughput • More initially but levels out • Increasing multiplexing decreases stream throughput • Balance need to complete ALL work vs. need to complete specific streams in available window • Will not present unreduced data for further tests

27. Run Time Predictions Based on multiplexing Benchmark • MPX=3 looks promising • Issue: Each file system must be on a single tape • Adding more tape drives will not reduce run time of individual streams • MPX=1 looks like best bet but need more drives to reduce total run time • Will running with more tape drives help?

28. Test Series 2 • Test effect of multiple streams • MPX=1 - No multiplexing • Same, Large HP-UX client • Expect 100Mbs link to saturate before client bus • Approximate link max throughput is around 12,000 KBS • Incremented # jobs 1 2 3 5 8 • Apparent saturation of link at 3 jobs

29. Multi-Streaming 100Mbs Link

30. Multi-Streaming 100Mbs Link

31. Multi-Streaming 100 Mbs Link

32. Observations • Multi-streaming effect similar to multiplexing • Link saturation at 3 streams • More than 3 streams resulted in significant degradation of stream throughput • Now we have useful information • We can predict runtime based on MPX and stream configurations

33. Run Time Predictions Based on Benchmark Results

35. Observations • No solution fits 8 hour window desired • MPX=1 and 3 Tapes/Streams is best solution • Adding more tape drives INCREASES total completion time • What looked like a simple problem is not so simple • Benchmarking revealed source of bottleneck • Recommended upgrade of link to Gb Ethernet

36. Benchmark of Gb Ethernet • Data not in paper due to timing of upgrade • Other test are in paper but not in this presentation • multiplexing benchmark • Multi-streaming benchmark • New run time predictions

37. Multiplexing Benchmark • Results similar to 1st test • MPX=3 gives best tape utilization • MPX bottleneck is not on the link

38. Run Time Predictions Based on MPX Test with 8 jobs on Gb Ethernet Link • In each case, the stream throughput dictates run time • Now we have a solution that solves the problem! • Solution based on 8 drives • How many do we really need?

39. Multi-Streaming Benchmark • Slight degradation of stream throughput • No apparent bottleneck on link • Only 8 drives available for test

40. Run Time Predictions Based on Benchmark Results

42. Observations • Solution fits 8 hour window desired • MPX=1 and 4 Tapes/Streams is best solution • For this workload we only need 4 drives to meet the window requirements. • What looked like a simple problem is not so simple • Benchmarking revealed source of bottleneck • Optimal solution uses minimum resources to accomplish objective • Applies to this workload but other workloads may require different solutions

43. Example Workload

44. Conclusions • Tuning a networked backup system is possible and necessary • Tuning requires understanding of the performance characteristics of YOUR environment • You have to understand your environment and the effect of available tuning options • Benchmarking is a useful tool for understanding performance characteristics of YOUR environment • Each configuration is different and simple ROT can produce undesirable results

45. Conclusions - continued • Benchmarking is not difficult • We used the reporting built into the NetBackup product • Tables, graphs were done with Excel • Not all combinations or systems have to be benchmarked • Benchmark important and/or representative workloads • Perform benchmark in steps • Prune the decision tree based on results of intermediate benchmarks

46. Conclusions - continued • Performing the benchmark is just the start • Reduce the data and present in different forms • In this example we presented same data in 4 different forms • Raw time-series data • Reduced and summarized data • Graph tuning options vs. performance results • Table to predict runtime impact of tuning choices • Each form was more useful and was the basis for the next

47. Conclusions - continued • The objective is not to maximize a single performance metric • The objective is to meet the requirements of your clients • Work completed • In available backup window • May have different requirements for different clients • One size does not fit all

48. Recommendations • DON’T use these numbers as ROT • Your environment will be different • Different workloads require different solutions • Do your own benchmarks • Analyze important and/or problematic workloads • Tune accordingly

49. Recommendations • Read the published papers • Write a paper about your experience • CMG mentor program • Get involved in local CMG region • Present at local meetings • Submit a paper for national CMG • Good way to get approval to attend! • You will benefit and your peers will benefit • New “short session” option at national CMG • Your peers do want to hear your experience

50. Resources • www.cmg.org • http://regions.cmg.org/regions • Bryan.Drake@VITA.VA.GOV • Questions?