400 likes | 597 Views
From application requests to Virtual IOPs: Provisioned key-value storage with Libra. David Shue * and Michael J. Freedman. (*now at Google). Shared Cloud. Tenant C. Tenant A. Tenant B. VM. VM. VM. VM. VM. VM. VM. VM. VM. Shared Cloud Storage. Tenant C. Tenant A. Tenant B. VM.
E N D
From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue* and Michael J. Freedman (*now at Google)
Shared Cloud Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM
Shared Cloud Storage Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM Block Storage Key-Value Storage SQL Database
Unpredictable Shared Cloud Storage Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM Key-Value Storage Block Storage Key-Value Storage SQL Database Disk IO-bound Tenants SSD-backed storage
Provisioned Shared Key-Value Storage ReservationA ReservationB ReservationC Tenant C Tenant A Tenant B VM VM VM VM VM VM VM VM VM Application Requests GET/s PUT/s (1KB normalized) Shared Key-Value Storage IOPs BW Low-level IO SSD SSD SSD SSD SSD
Libra Contributions • Libra IO Scheduler • Provisions low-level IOallocations for app-requestreservations w/ high utilization. • Supports arbitrary object distributions and workloads. • 2 key mechanisms • Track per-tenant app-request resource profiles. • Model IO resources with Virtual IOPs.
Tenant B VM VM VM Provisioned Distributed Key-Value Storage Global Reservation Problem [Pisces OSD1 ’12] ReservationA ReservationB Tenant A Tenant B VM VM VM VM VM VM Shared Key-Value Storage Local Reservation Problem Storage Node 1 Storage Node N ...
Tenant A Tenant B VM VM VM VM VM VM Provisioned Distributed Key-Value Storage ReservationA ReservationB demand data partitions Storage Node 1 Storage Node N ...
Provisioned Distributed Key-Value Storage ReservationA ReservationA ReservationB ReservationB Storage Node 1 Storage Node N ...
Provisioned Distributed Key-Value Storage ReservationB ReservationB ReservationA ReservationA ResA1 ResB1 ResAn ResBn Storage Node 1 Storage Node N ...
Provisioned Local Key-Value Storage GET 1001100 GET K Key-Value Protocol Retrieve K Persistence Engine Read l337 Libra IO Scheduler IO Scheduler IO operation Physical Disk
GET PUT Libra Design Persistence Engine Reservation Distribution Policy Provision IO allocations for tenant app-request reservations blah Libra IO Scheduler Determine the cost of a tenant request Libra Provisioning Policy How much IO to provision? How much IO to consume? DRR blah Physical Disk
Provisioning App-request Reservations is Hard IO Amplification 1 KB PUT ≥ 1 KB Write Track tenant app-request resource profiles ReservationBn IO Interference Underestimate provisionable IO Variable IO throughput Non-linear IO Performance Model IO with Virtual IOPs Non-linear cost per KB
H F O M V index V2 V3 V3 A O A index index V1 V0 index V3 Workload-dependent IO Amplification LevelDB (LSM-Tree) PUT K,V3 PUT FLUSH COMPACT
H F O M V index V2 V3 V3 A O A index index V1 V0 index V3 Workload-dependent IO Amplification LevelDB (LSM-Tree) PUT K,V3 PUT FLUSH COMPACT
H F O M V index V2 V3 V3 A O A index index V1 V0 index V3 Workload-dependent IO Amplification LevelDB (LSM-Tree) PUT K,V3 PUT FLUSH COMPACT
H F O M V A G Z index index index index K Workload-dependent IO Amplification GET K Workload-dependent IO Amplification
GET PUT IO IO Libra Tracks App-request IO Consumption to Determine IO Allocations Tenant A 500 IO/s Libra IO Scheduler blah Compute app-request IO profiles Track IO consumption Provision IO allocations Libra Provisioning Policy 5 5 GET EWMA per-request averages 80 100 x Per-GET 1 25 PUT = + 1 x 50 Per-PUT 6 FLUSH 0.5 PUT Tenant A 5 IO units 70 500 blah COMPACT FLUSH
Unpredictable IO Interference 4 read/4 write tenants • Die-level parallelism, low latency IOPs • Shared-controller and bus contention • Erase-before-write overhead • FTL and read-modify-write garbage colleciton
Unpredictable IO Interference Unpredictable IO Interference 21
Libra Underestimates IO Capacity to Ensure Provisionable Throughput Provisionable IO throughput = floor(workloads) (18 Kop/s) Provisionable IO limit
Libra Underestimates IO Capacity to Ensure Provisionable Throughput Provisionable IO throughput = floor(workloads) (18 Kop/s) Provisionable IO limit
Libra Underestimates IO Capacity to Ensure Provisionable Throughput Provisionable IO throughput = floor(workloads) (18 Kop/s) Provisionable IO limit
Non-linear IO Performance Max BW Max IOP/s IOP Cost = linear decrease until 6 KB, then constant IOP Cost = constant until 6 KB, then linear increase
Non-linear IO Performance Non-linear IO Performance Max BW Max IOP/s
Libra Uses Virtual IOPs to Model IO Resources • Unifies IO cost into a single metric • Captures non-linear IO performance • Provides IO insulation 2 equal-allocation tenants IO Insulation = 1/2 Max Read/Write
Libra Uses Virtual IOPs to Model IO Resources 2 equal-allocation tenants IO Insulation = 1/2 Max Read/Write
Libra Design Persistence Engine Update tenant VOP allocations Provision IO allocations for tenant app-request reservations blah Libra IO Scheduler Determine the cost of a tenant request Libra Provisioning Policy Charge tenant IOPs based on VOP cost Provision VOPs within provisionable limit blah Track app-request VOP consumption Physical Disk
Evaluation • Does Libra's IO resource model achieve accurate resource allocations? • Does Libra's IO threshold make an acceptable tradeoff of performance for predictability in a real storage stack? • Can Libra ensure per-tenant app-request reservations while achieving high utilization?
Libra Achieves Accurate IO Allocations Interference-free Ideal even Read 1 KB Throughput Ratio = Actual / Expected (IO Insulation)
Libra Achieves Accurate IO Allocations Interference-free Ideal Write 1-256 KB Throughput Ratio = Actual / Expected (IO Insulation)
Libra Achieves Accurate IO Allocations Min-Max Ratio = Min Throughput Ratio / Max Throughput Ratio
Libra Trades-off Nominal IO Throughput For Predictability Unprovisionable Throughput As a Percentage of Total Throughput < 10th percentile covered by SLA and higher-level policies
Libra Achieves App-request Reservations 0.5x 1.5x Read Heavy Mixed Write Heavy Fully provisioned allocations Work-conserving consumption of unprovisioned resources
Libra Achieves App-request Reservations Read Heavy Mixed Write Heavy
Conclusion • Libra IO Scheduler • Provisions IOallocations for app-requestreservations w/ high utilization. • Supports arbitrary object distributions and workloads. • 2 key mechanisms • Track per-tenant app-request resource profiles. • Model IO resources with Virtual IOPs. • Evaluation • Achieves accurate low-level IO allocations. • Provisions the majority of IO resources over a wide range of workloads • Satisfies app-request reservations w/ high utilization.