1 / 32

Aurora

A new model and architecture for data stream management. Aurora. Data Stream Management. Why on earth would one need it?. The Problem: Tokyo Traffic Control . Stream Processing for Traffic Control. 24-hour real-time control 1.000 traffic intersections 15.154 traffic signals Input Cameras

irish
Download Presentation

Aurora

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A new model and architecture for data stream management Aurora

  2. Data Stream Management Why on earth would one need it?

  3. The Problem: Tokyo Traffic Control

  4. Stream Processing for Traffic Control • 24-hour real-time control • 1.000 traffic intersections • 15.154 traffic signals • Input • Cameras • Helicopters • Police • Citizen reports • 17.000 vehicle detectors • Onboard vehicle sensors • Traffic jams, accidents & closed streets • Output • Central monitors • 300 traffic information boards • Digital speed signs • Route signs • Affectors • Adjusted traffic signal lights (7.000) • Communications with officers on site

  5. TTC: Center Display Board

  6. TTC: Information Board

  7. Other Example Domains • Smart Energy Grid Management • Network Traffic Management • Targeted Adversating • System Monitoring • Military Logistics • Online Auctions • Habitat Monitoring • Immersive Environments

  8. Stream Processing Engines • HADP vs DAHP • Continuous Queries • Real-time processing • Transient data • Lossy information

  9. Aurora Overview

  10. The Topic • Aurora • The prototype • DBMS / SPE / DSMS • UI • The query language • The project • The authors

  11. The Authors • M.I.T. , Department of EECS and Laboratory of Computer Science • Michael Stonebraker • Brandeis University, Department of Computer Science • Daniel J. Abadi • Mitch Cherniack • Brown University , Department of Computer Science • Don Carney • Uğur Çetintemel • Christian Convey • Sangdon Lee • Nesime Tatbul • Stan Zdonik

  12. Talk Overview • Why would you need it? • Who did it? • Stream Processing Engines in general • SQuAl • Operators • Runtime • Scheduling , QOS and Optimization • Related work

  13. Aurora SQuAl (Stream Query Algebra)

  14. SQuAl Overview • Connection Points • Models • Continuous Query • View • Ad-hoc Query • Operators • Order-agnostic • Order-sensitive

  15. SQuAl Operators • Order-agnostic • Filter • Map • Union • Order-sensitive • BSort • Aggregate • Join • Resample • Quirks!

  16. Union (Unordered)

  17. BSort (Ordered)

  18. SQuAl: Example

  19. Aurora Runtime

  20. Real-time Scheduling • Timestamped Tuples • Train scheduling • Interbox nonlinearities • Intrabox nonlinearities • Superboxes • Introspection • Static • Dynamic

  21. Handling overload • QoS specifications • Response times • Tuple drops • Values produced • Load Shedding • Not Implemented at the time

  22. Query Optimization • Dynamic Continuous Query Optimization • Inserting projections (map) • Combining boxes • Reordering boxes • Ad-hoc query optimization

  23. Aurora Related work

  24. Related work • STREAM • Stanford University, 2000-2006 • Telegraph • UC Berkley, 2000-2007? • SASE • UC Berkley / Mass Amherst, 2006-2008? • Cayuga • Cornell University, 2005-2007? • PIPES • University of Marburg, 2003-2007? • NiagaraCQ • University of Wiscon-Madison, 1999-2002

  25. Aurora’s Evolution

  26. Complex Event Processing Today • Oracle • Oracle CEP • Microsoft • MS SQL Server StreamInsight • Open Source • OpenPDC • Aleri • Coral8 • TruViso • StreamBase • Aurora’s Grandchild • IBM • SPADE • Active Middleware Technology

  27. Summary, Questions? • SPEs address different problems • e.g. dynamic realtime monitoring • Data Active, Human Passive • Realtime, transient, even lossy data • Aurora evolved into StreamBase • SQuAl evolved into StreamSQL • Many production-quality alternatives

  28. Filter (Unordered)

  29. Map (Unordered)

  30. Aggregate (Ordered)

  31. Join (Ordered)

  32. Resample (Ordered) • Based on RRDTool’s philosophy? • Paper: • Simple interpolation • Use The Force, Read The Source: • Average • Count • Sum • Max • Min • LastVal

More Related