1 / 16

Reducing Execution Overhead in a Data Stream Manager

Reducing Execution Overhead in a Data Stream Manager. Don Carney Brown University Uğur Çetintemel Brown University Mitch Cherniack Brandeis University Alex Rasin Brown University Michael Stonebraker MIT Stan Zdonik Brown University. App. App. App. QoS. QoS. QoS.

emares
Download Presentation

Reducing Execution Overhead in a Data Stream Manager

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reducing Execution Overhead in a Data Stream Manager Don Carney Brown University Uğur ÇetintemelBrown University Mitch Cherniack Brandeis University Alex Rasin Brown University Michael Stonebraker MIT Stan Zdonik Brown University MPDS 2003 San Diego

  2. App App App QoS QoS QoS Aurora from the Sky Queries . . . . . . . . . . . . . . . MPDS 2003 San Diego

  3. App App App QoS QoS QoS Aurora from the Sky . . . . . . . . . . . . . . . MPDS 2003 San Diego

  4. inputs outputs Storage Manager q1 q2 . . . s s qi m Buffer . . . . . . È È Persistent Store Catalog q1 q2 . . . qn … … … … … … Runtime OperationBasic Architecture Router Scheduler Box Processors QOS Monitor MPDS 2003 San Diego

  5. Execution Model • Traditional Thread-driven Execution • Thread per query or operator • Resource management done by OS • Easy to program • Scalability problems • State-based Execution • Single scheduler thread maintains execution queue • Small number of worker threads execute execution queue entries • Enables application specific allocation of resources MPDS 2003 San Diego

  6. State-Based vs. Thread-Based MPDS 2003 San Diego

  7. Scheduling • Two level scheduling • Inter-query scheduling (Which query?) • Intra-query scheduling (Operation order?) • Batching • Tuple trains • Fewer box executions -> fewer scheduling decisions • Also, better memory utilization • Superbox scheduling • Multiple boxes per decision -> fewer scheduling decisions • Memory utilization: allocate for entire superbox at once • State Monitoring (# tuples, latencies, etc) • Incremental and approximate MPDS 2003 San Diego

  8. … … … z z z y y y x x x AB B (A (z)) B (A (y)) B (A (x)) Box Trains: B B (A (z), A (y), A (x)) A A (z, y, x) Tuple Trains: Runtime OperationScheduling: Minimizing Per Tuple Processing Overhead Train Scheduling: B A A (z) A (y) A (x) B (A (z)) B (A (y)) B (A (x)) = Scheduler Action MPDS 2003 San Diego

  9. Tuple Trains and Superboxes MPDS 2003 San Diego

  10. Overheads MPDS 2003 San Diego

  11. Overheads MPDS 2003 San Diego

  12. Other Issues • Priority assignment • Box Execution Order • QoS MPDS 2003 San Diego

  13. Stay Tuned! • SIGMOD Demo • VLDB ’03 paper “Operator Scheduling in a Data Stream Environment” MPDS 2003 San Diego

  14. App App QoS QoS A little closer . . . . . . . . . . . . MPDS 2003 San Diego

  15. App App QoS QoS A little closer . . . . . . . . . . . . MPDS 2003 San Diego

  16. App App QoS QoS Aurora from the Sky Query . . . . . . . . . . . . Query MPDS 2003 San Diego

More Related