1 / 32

Performance Isolation and Fairness for Multi-Tenant Cloud Storage

Performance Isolation and Fairness for Multi-Tenant Cloud Storage. David Shue *, Michael Freedman*, and Anees Shaikh ✦. *Princeton ✦ IBM Research. Setting: Shared Storage in the Cloud. Y. Y. F. F. Z. Z. T. T. Shared Key-Value Storage. S3. EBS. SQS. Predictable Performance is Hard.

rusty
Download Presentation

Performance Isolation and Fairness for Multi-Tenant Cloud Storage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance Isolation and Fairness for Multi-Tenant Cloud Storage David Shue*, Michael Freedman*, and Anees Shaikh✦ *Princeton ✦IBM Research

  2. Setting: Shared Storage in the Cloud Y Y F F Z Z T T Shared Key-Value Storage S3 EBS SQS

  3. Predictable Performance is Hard Z Z Z Y Y Y Y T T F F F F F Shared Key-Value Storage DD DD DD DD Multiple co-located tenants ⇒ resource contention

  4. Predictable Performance is Hard Z Z Z Y Y Y Y T T F F F F F Fair queuing @ big iron DD Shared Key-Value Storage DD DD DD DD DD DD DD Multiple co-located tenants ⇒ resource contention

  5. Predictable Performance is Hard Z Z Z Y Y Y Y T T F F F F F SS SS SS SS Multiple co-located tenants ⇒ resource contention Distributed system ⇒ distributed resource allocation

  6. Y F T Z keyspace keyspace keyspace keyspace Predictable Performance is Hard data partition popularity Z Z Z Y Y Y Y T T F F F F F SS SS SS SS Multiple co-located tenants ⇒ resource contention Distributed system ⇒ distributed resource allocation

  7. Predictable Performance is Hard GET SET SET GET 1kB 10B 10B 1kB (large reads) (small reads) (large writes) (small writes) Z Z Z Z Z Z Y Y Y Y Y Y Y Y T T T T F F F F F F F F F F SS SS SS SS Multiple co-located tenants ⇒ resource contention Distributed system ⇒ distributed resource allocation Skewed object popularity ⇒ variable per-node demand Disparate workloads ⇒ different bottleneck resources

  8. Tenants Want System-wide Resource Guarantees demandz = 120 kreq/s 80 kreq/s 120 kreq/s 160 kreq/s 40 kreq/s ratez = 80 kreq/s Zynga Yelp TP Foursquare demandf = 120 kreq/s ratef = 120 kreq/s Z Z Z Y Y Y Y T T F F F F F Hard limits ⇒ lower utilization Shared Key-Value Storage SS SS SS SS Multiple co-located tenants ⇒ resource contention Distributed system ⇒ distributed resource allocation Skewed object popularity ⇒ variable per-node demand Disparate workloads ⇒ different bottleneck resources

  9. Pisces Provides Weighted Fair-shares wz = 20% wy = 30% wt = 10% wf = 40% Zynga Yelp TP Foursquare demandz = 30% Z Z Z Y Y Y Y T T F F F F F ratez = 30% demandf = 30% ratef = 30% Shared Key-Value Storage Max-min fair share ⇒ lower bound on system performance SS SS SS SS Multiple co-located tenants ⇒ resource contention Distributed system ⇒ distributed resource allocation Skewed object popularity ⇒ variable per-node demand Disparate workloads ⇒ different bottleneck resources

  10. Pisces: Predictable Shared Cloud Storage • Pisces • Per-tenant max-min fair shares of system-wide resources ~ min guarantees, high utilization • Arbitrary object popularity • Different resource bottlenecks • AmazonDynamoDB • Per-tenant provisioned rates ~ rate limited, non-work conserving • Uniform object popularity • Single resource (1kB requests)

  11. RS FQ PP Predictable Multi-Tenant Key-Value Storage Tenant A Tenant B VM VM VM VM VM VM GET 1101100 Controller RR

  12. WA PP FQ RS WA2WB2 Predictable Multi-Tenant Key-Value Storage WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM GET 1101100 Controller RR WA2 WB2 WA1 WB1

  13. PP RS FQ WA WA2WB2 Strawman: Place Partitions Randomly WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM Controller RR

  14. WA FQ RS PP WA2WB2 Strawman: Place Partitions Randomly WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM Controller Overloaded RR

  15. WA PP RS FQ WA2WB2 Pisces: Place Partitions By Fairness Constraints WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM Collect per-partition tenant demand Controller RR Bin-pack partitions

  16. WA FQ RS PP WA2WB2 Pisces: Place Partitions By Fairness Constraints WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM Controller RR Results in feasible partition placement

  17. RS WA FQ PP WA2WB2 Strawman: Allocate Local Weights Evenly WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM Controller excessive demand Overloaded RR WA1 =WB1 WA2 =WB2

  18. FQ WA PP RS WA2WB2 Pisces: Allocate Local Weights By Tenant Demand WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM Compute per-tenant +/- mismatch max mismatch Controller M/M/1 queuing delay RR WA1 > WB1 WA1 =WB1 WA2 < WB2 WA2 =WB2 Reciprocal weight swap A←B A→B

  19. FQ WA RS PP WA2WB2 Strawman: Select Replicas Evenly WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM GET 1101100 Controller 50% 50% RR WA1 > WB1 WA2 < WB2

  20. WA PP RS FQ WA2WB2 Pisces: Select Replicas By Local Weight WeightA WeightB Tenant A Tenant B VM VM VM VM VM VM GET 1101100 Controller detect weight mismatch by request latency 50% 40% 60% 50% RR WA1 > WB1 WA2 < WB2

  21. FQ WA PP RS WA2WB2 out req out req Strawman: Queue Tenants By Single Resource Tenant A Tenant B VM VM VM VM VM VM Bandwidth limited Request Limited Controller bottleneck resource (out bytes) fair share RR GET 0100111 GET 1101100 WA1 > WB1 WA2 < WB2

  22. FQ WA PP RS WA2WB2 out req out req Pisces: Queue Tenants By Dominant Resource Tenant A Tenant B VM VM VM VM VM VM Bandwidth limited Request Limited Controller dominant resource fair share RR bottlenecked by out bytes Track per-tenant resource vector WA2 < WB2

  23. FQ WA PP RS WA2WB2 Weight Allocations Replica Selection Policies Pisces Mechanisms Solve For Global Fairness Maximum bottleneck flow weight exchange global feasible partition placement Controller FAST-TCP basedreplica selection demand-driven weight allocation fairness and capacity constraints RR System Visibility weight-sensitive selection policy ... RR DRR token-basedDRFQ scheduler dominant resource fair shares SS local ... SS microseconds seconds minutes Timescale

  24. Evaluation • Does Pisces achieve (even) system-wide fairness? • Is each Pisces mechanism necessary for fairness? • What is the overhead of using Pisces? • Does Pisces handle mixed workloads? • Does Pisces provide weighted system-wide fairness? • Does Pisces provide local dominant resource fairness? • Does Pisces handle dynamic demand? • Does Pisces adapt to changes in object popularity?

  25. Evaluation • Does Pisces achieve (even) system-wide fairness? • Is each Pisces mechanism necessary for fairness? • What is the overhead of using Pisces? • Does Pisces handle mixed workloads? • Does Pisces provide weighted system-wide fairness? • Does Pisces provide local dominant resource fairness? • Does Pisces handle dynamic demand? • Does Pisces adapt to changes in object popularity?

  26. Pisces Achieves System-wide Per-tenant Fairness Ideal fair share: 110 kreq/s (1kB requests) Unmodified Membase Pisces 0.57 MMR 0.98 MMR 8 Tenants - 8 Client - 8 Storage Nodes Zipfian object popularity distribution Min-Max Ratio: min rate/max rate (0,1]

  27. WA PP RS RS FQ WA2WB2 Each Pisces Mechanism Contributes to System-wide Fairness and Isolation FQ RS FQ PP FQ PP WA WA Unmodified Membase 0.57 MMR 0.59 MMR 0.93 MMR 0.98 MMR 0.64 MMR 0.90 MMR 2x vs 1x demand 0.58 MMR 0.74 MMR 0.96 MMR 0.89 MMR 0.97 MMR 0.36 MMR

  28. Pisces Imposes Low-overhead > 19% < 5%

  29. Pisces Achieves System-wide Weighted Fairness 0.98 MMR 0.89 MMR 0.91 MMR 4 heavy hitters 20 moderatedemand 40 lowdemand 0.91 MMR 0.56 MMR

  30. Pisces Achieves Dominant Resource Fairness 1kB workload bandwidth limited 10B workload request limited 76% of bandwidth 76% of request rate GET Requests (kreq/s) Bandwidth (Mb/s) 24% of request rate Time (s) Time (s)

  31. Pisces Adapts to Dynamic Demand Tenant Demand Constant Diurnal (2x wt) Bursty even ~2x

  32. Partition Placement Weight Allocation Replica Selection Fair Queuing PP WA RS FQ Conclusion • PiscesContributions • Per-tenant weighted max-min fair shares of system-wide resources w/ high utilization • Arbitrary object distributions • Different resource bottlenecks • Novel decomposition into 4 complementary mechanisms

More Related