1 / 31

Flux

Flux. Flux: An Adaptive Partitioning Operator for Continuous Query Systems M.A. Shah, J.M. Hellerstein, S. Chandrasekaran, M.J. Franklin UC Berkeley Presenter: Bradley Momberger. Overview. Introduction Background Experiments and Considerations Conclusion. Introduction.

evictoria
Download Presentation

Flux

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Flux Flux: An Adaptive Partitioning Operator for Continuous Query Systems M.A. Shah, J.M. Hellerstein, S. Chandrasekaran, M.J. Franklin UC Berkeley Presenter: Bradley Momberger

  2. Overview • Introduction • Background • Experiments and Considerations • Conclusion

  3. Introduction • Continuous query (CQ) systems • Create unbounded, streaming results from unbounded, streaming data sources. • May in the long run have scalability issues, due to the need for fast response times, the possibility of large numbers of users, and the management of potentially large histories. • Are only as fast as their constituent operators will allow.

  4. Parallelism • Traditional parallelism techniques • Poor fit for CQ systems • Not adaptive • CQ requires adaptability to changing conditions

  5. Overview • Introduction • Background • Experiments and Considerations • Conclusion

  6. Background • Exchange • Producer-consumer pair • Ex-Prod: Intermediate producer instance connected to consumers • Ex-Cons: Intermediate consumer instance which polls inputs from all producers. • “Content sensitive” routing • RiverDQ • “Content insensitive” routing • Random choice of Ex-Cons target

  7. Flux • Flux, Fault-tolerant Load-balancing eXchange • Load balancing through active repartitioning • Producer-consumer pair • Buffering and reordering • Detection of imbalances

  8. Short Term Imbalances • A stage runs only as fast as its slowest Ex-Cons • Head-of-line blocking • Uneven distribution over time • The Flux-Prod solution • Transient Skew buffer • Hashtable buffer between producer and Flux-Prod • Get new tuples for each Flux-Cons as buffer space becomes available. • On-demand input reordering

  9. Flux-Prod Design

  10. Long Term Imbalances • Eventually overload fixed size buffers • Cannot use same strategy as short term • The Flux-Cons solution • Repartition at consumer level • Move states • Aim for maximal benefit per state moved • Avoid “thrashing”

  11. Flux-Cons Design

  12. Memory Constrained Environment • First tests were done with adequate memory • Does not necessarily reflect reality • Memory shortages • Large histories • Extra operators • Load shedding with little memory • Push to disk • Move to other site • Decrease history size • May not be acceptable in some applications

  13. Flux and Constrained Memory • Dual-destination repartitioning • Other machines • Disk storage • Local mechanism • Flux-Cons spills to disk when memory is low • Retrieves from disk when memory becomes available • Global Memory Constrained Repartitioning • Poll Flux-Cons operators for memory usage • Repartition based on results

  14. Memory-Adaptive Flux-Cons

  15. Overview • Introduction • Background • Experiments and Considerations • Conclusion

  16. Experimental Methodology • Example operator • Hash-based, windowed group-by-aggregate • Statistic over fixed-size history • Cluster hardware • CPU: 1000 MIPS • 1GB main memory • Network simulation • 1K packet size, infinite bandwidth, 0.07ms latency • Virtual machines, simulated disk.

  17. Experimental Methodology • Simulator • TelegraphCQ base system • Operators share physical CPU with event simulator • Aggregate evaluation and scheduler simulated • Testbed • Single producer-consumer stage • 32 nodes in simulated cluster • Ex-Cons operator dictates performance

  18. Short Term Imbalance Experiment • Give Flux stage a transient skew buffer • Compare to base Exchange stage with equivalent space • Comparison statistics • 500ms load per virtual machine, round robin • Simulated process: 0.1ms processing, 0.05ms sleep • 16s runtime (32 machines  0.5s/machine)

  19. Short Term Imbalance Experiment

  20. Long Term Imbalance Experiment • Operator stage • 64 partitions per virtual machine • 10,000 tuple (800KB) history per partition • 160KB skew buffer • 0.2μs per tuple for partition processing • Network • 500mbps throughput for partitions • 250mbps point-to-point

  21. Balancing Processing Load

  22. Graceful Degradation

  23. Varying Collection Time

  24. Memory Constrained Experiments • Memory “pressure” • 768MB initial memory load • 6MB/partition  128 partitions/machine • Available memory => 512MB (down from 1GB) • Change made after 1s of simulation • 14s required to push the remaining 256MB • May be to disk or to other machines

  25. Throughput during Memory Balancing

  26. Avg. Latency during Memory Balancing

  27. Average Latency Degradation

  28. Hybrid Policy • Combines previous policies • Memory-based policy when partitions are on disk • Minimize latency • Load-balancing policy when all partitions are in memory • Maximize throughput

  29. Comparative Review ┌ Steady state ┐ ┌ last 20 seconds of simulation ┐

  30. Overview • Introduction • Background • Experiments and Considerations • Conclusion

  31. Conclusions • Flux • Is a reusable mechanism • Encapsulates adaptive repartitioning • Extends the Exchange operator • Alleviates short- and long-term imbalances • Outperforms static partitioning when correcting imbalances • Can use hybrid policies to adapt to changing processing and memory requirements.

More Related