1 / 34

Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services. Ke Wang , Abhishek Kulkarni , Mi chael Lang, Dorian Arnold, Ioan Raicu USRC @ Los Alamos National Laboratory Datasys @ Illinois Institute of Technology CS @ Indiana University

fay
Download Presentation

Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services Ke Wang, AbhishekKulkarni, Michael Lang, Dorian Arnold, IoanRaicu USRC @ Los Alamos National Laboratory Datasys @ Illinois Institute of Technology CS @ Indiana University CS @ University of New Mexico November 20th, 2013 at IEEE/ACM Supercomputing/SC 2013

  2. Current HPC System Services 2 Extreme scale Lack of decomposition for insight Many services have centralized designs Impacts of service architectures  an open question Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  3. Long Term Goals • Modular components design for composable services • Explore the design space for HPC services • Evaluate the impacts of different design choices Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  4. Contribution • A taxonomy for classifying HPC system services • A simulation tool to explore Distributed Key-Value Stores (KVS) design choices for large-scale system services • An evaluation of KVS design choices for extreme-scale systems using both synthetic and real workload traces Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  5. Outline • Introduction & Motivation • Key-Value Store Taxonomy • Key-Value Store Simulation • Evaluation • Conclusions& Future Work Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  6. Outline • Introduction & Motivation • Key-Value Store Taxonomy • Key-Value Store Simulation • Evaluation • Conclusions& Future Work Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  7. Distributed System Services • Job Launch, Resource Management Systems • System Monitoring • I/O Forwarding, File Systems • Function Call Shipping • Key-Value Stores Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  8. Key IssuesDistributed System Services • Scalability • Dynamicity • Fault Tolerance • Consistency Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  9. Key-Value Stores and HPC • Large volume of data and state information • Distributed NoSQL data stores used as building blocks • Examples: • Resource management (job, node status info) • Monitoring (system active logs) • File systems (metadata) • SLURM++, MATRIX [1], FusionFS [2] [1] K. Wang, I. Raicu. “Paving the Road to exascale through Many Task Computing”, Doctor Showcase, IEEE/ACM Supercomputing 2012 (SC12) [2] D. Zhao, I. Raicu. “Distributed File Systems for Exascale Computing”, Doctor Showcase, IEEE/ACM Supercomputing 2012 (SC12) Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  10. Outline • Introduction & Motivation • Key-Value Store Taxonomy • Key-Value Store Simulation • Evaluation • Conclusions& Future Work Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  11. HPC KVS TaxonomyWhy? • Decomposition • Categorization • Suggestion • Implication Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  12. HPC KVS TaxonomyComponent • Service model: functionality • Data model: distribution and management of data • Network model: dictates how the components are connected • Recovery model: how to deal with component failures • Consistency model: how rapidly data modifications propagate Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  13. Centralized Architectures Data model: centralized Network model: aggregation tree Recovery model: fail-over Consistency model: strong Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  14. Distributed Architectures Data Model: distributed with partition Network Model: fully-connected partial knowledge Recovery Model: consecutive replicas Consistency Model: strong, eventual Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  15. Outline • Introduction & Motivation • Key-Value Store Taxonomy • Key-Value Store Simulation • Evaluation • Conclusions& Future Work Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  16. KVS Simulation Design • Discrete Event Simulation  PeerSim • Evaluated others: OMNET++, OverSim, SimPy • Configurable number of servers and clients • Different architectures • Two parallel queues in a server • Communication queue (send/receive requests) • Processing queue (process request locally) Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  17. Simulation Cost Model The time to resolve a query locally (tLR), and the time to resolve a remote query (tRR) is given by: tLR= CS + SR + LP + SS + CR For fully connected: tRR= tLR+ 2 × (SS + SR) For partially connected: tRR= tLR+ 2k× (SS + SR) where k is the number of hops to find the predecessor Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  18. Failure/Recovery Model • Defines what to do when a node fails • How a node-state recovers when rejoining after failure EM EM client client X X notify back notify failure s0 r5,1 r4,2 s0 r5,1 r4,2 s1 r0,1 r5,2 s1 r0,1 r5,2 s5 r4,1 r3,2 s5 r4,1 r3,2 s0, s4, s5 data first replica down s0 is back replicate my data remove s5data second replica down s4 r3,1 r2,2 s4 r3,1 r2,2 s0 is back s2 r1,1 r0,2 s2 r1,1 r0,2 replicate my data replicate s0 data remove s0 data s3 r2,1 r1,2 s3 r2,1 r1,2 Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  19. Consistency Model • Strong Consistency • Every replica observes every update in the same order • Client sends requests to a dedicated server (primary replica) • Eventual Consistency • Requests are sent to randomly chosen replica (coordinator) • Three key parameters: N, R, W, satisfying R + W > N • Use Dynamo [G. Decandia, 2007] version clock to track different versions of data and detect conflicts Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  20. Outline • Introduction & Motivation • Key-Value Store Taxonomy • Key-Value Store Simulation • Evaluation • Conclusions& Future Work Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  21. Evaluation • Evaluate the overheads • Different architectures, focus on distributed ones • Different models • Light-weight simulations: • Largest experiments  25GB RAM, 40 min walltime • Workloads • Synthetic workload with 64-bit key space • Real workload traces from 3 representative system services: job launch, system monitoring, and I/O forwarding Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  22. Validation • Validate against ZHT [1] (left) and Voldemort (right) • ZHT  BG/P up to 8K nodes (32K cores) • Voldemort  PROBE Kodiak Cluster up to 800 nodes [1] T. Li, X. Zhou, K. Brandstatter, D. Zhao, K. Wang, A. Rajendran, Z. Zhang, I. Raicu. “ZHT: A Light-weight Reliable Persistent Dynamic Scalable Zero-hop Distributed Hash Table”, IEEE International Parallel & Distributed Processing Symposium (IPDPS) 2013 Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  23. Fully-connected vsPartial-connected • Partial connectivity  higher latency due to the additional routing • Fully-connected topology  faster response (twice as fast at extreme scale) Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  24. Replication Overhead • Adding replicas always involve overheads • Replicas have larger impact on fully connected than on partially connected Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  25. Failure Effect • Higher failure frequency introduces more overhead, but the dominating factor is the client request processing messages Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  26. Combined Overhead • Eventual consistency has more overhead than the strong consistency Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  27. Real Workloads • For job launch and I/O forwarding • Eventual consistency performs worse  almost URD for both request type and the key • Monitoring • Eventual consistency works better  all requests are “put” Fully connected Partially connected Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  28. Simulation  Real Services • ZHT (distributed key/value storage) • DKVS implementation • MATRIX (runtime system) • DKVS is used to keep task meta-data • SLURM++ (job management system) • DKVS is used to store task & resource information • FusionFS (distributed file system) • DKVS is used to maintain file/directory meta-data Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  29. Outline • Introduction & Motivation • Key-Value Store Taxonomy • Key-Value Store Simulation • Evaluation • Conclusions& Future Work Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  30. Conclusions • Key-value Store is building block • Service taxonomy is important • Simulation framework to study services • Distributed architecture is demanded • Replication adds overhead • Fully-connected topology is good • As long as the request processing message dominates • Consistency tradeoffs Read-Intensity/Performance Weak Consistency Strong Consistency Eventual Consistency Write-Intensity/Availability Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  31. Future Work • Extend the simulator to cover more of the taxonomy • Explore other recovery models • log-based • information dispersal algorithm • Explore other consistency models • Explore using DKVS in the development of: • General building block library • Distributed monitoring system service • Distributed message queue system Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  32. Acknowledgement • DOE contract: DE-FC02-06ER25750 • Part of NSF award: CNS-1042543 (PRObE) • Collaboration with FusionFSproject under NSF grant: NSF-1054974 • BG/P resource from ANL • Thanks to Tonglin Li, Dongfang Zhao, HakanAkkan Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  33. More Information • More information: • http://datasys.cs.iit.edu/~kewang/ • Contact: • kwang22@hawk.iit.edu • Questions? Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

  34. Related Work • Service Simulation • Peer-to-peer networks simulation • Telephony simulations • Simulation of consistency • Problem: not focus on HPC, or combine distributed features • Taxonomy • Investigation of distributed hash tables, and an algorithm taxonomy • Grid computing workflows taxonomy • Problems: none of them drive features in a simulation Using Simulation to Explore Distributed Key-Value Stores for Extreme-Scale System Services

More Related