1 / 27

Microsoft StreamInsight

Microsoft StreamInsight. Allan Mitchell Konesans Limited. Who am I?. Allan Mitchell Joint author on “First to Market” SSIS Book by Wrox Websites www.SQLIS.com www.SQLDTS.com www.konesans.com StreamInsight and SQL Azure Advisory Boards Microsoft SQL Server MVP Twitter : allanSQLIS

tia
Download Presentation

Microsoft StreamInsight

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Microsoft StreamInsight Allan Mitchell Konesans Limited

  2. Who am I? • Allan Mitchell • Joint author on “First to Market” SSIS Book by Wrox • Websites • www.SQLIS.com • www.SQLDTS.com • www.konesans.com • StreamInsight and SQL Azure Advisory Boards • Microsoft SQL Server MVP • Twitter: allanSQLIS • eMail: allan.mitchell@konesans.com

  3. Agenda • What is Stream Processing • Understanding Streaming Data • Differences to RDBMS • Scenarios • StreamInsight Architecture • Concepts • Typical Patterns

  4. What is Event Stream Processing Query • Database loads data from disk • You query database • Data is stale

  5. What is Stream Processing Query • Continuous processing of event streams • from multiple sources • based on a declarative query language • with near-zero latency StreamInsight

  6. Understanding Streaming Data Using an RDBMS: Q. Given a time interval tell me the amount of Electricity units used per household per time interval from now until I say stop. Not Easy Requires requerying/polling Need to store the end of the last interval

  7. Understanding Streaming Data Answering with StreamInsight • Create a window over the event stream • Partition/Group By Household • SUM() readings within window This is the streaming data paradigm in a nutshell – ask questions about data in flight.

  8. What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

  9. What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

  10. What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

  11. What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

  12. Latency Scenarios for Event-Driven Applications Relational Database Applications StreamInsight Target Scenarios Operational Analytics Applications, e.g., Logistics, etc. Data Warehousing Applications Web Analytics Applications Manufacturing Applications Financial trading Applications Monitoring Applications Aggregate Data Rate (Events/sec.)

  13. Who might need CEP • Fraud Detection • Real-Time Trade Risking • Algorithmic Trading/Betting • Meter throughputs • Oil, Gas, Water, Electricity • Use to drive alarms, alerts etc • Telemetry Data

  14. Windowing • Hopping • Tumbling • Snapshot • Count

  15. Time Windows Hopping Window Time

  16. Time Windows Tumbling Window Time

  17. Snapshot Windows

  18. Snapshot Windows

  19. StreamInsight Key Features • Runtime environment for continuous and event-driven processing • In-memory processing, no persistence • Declarative query language: LINQ • .NET APIs for Input/Output Adapters • Flexible deployment models • Extensibility: Aggregates, Operators • Rich diagnostic & debugging interface

  20. StreamInsight Platform StreamInsightApplication Development StreamInsight Application at Runtime Event sources Event targets Input Adapters Output Adapters Devices, Sensors Pagers & Monitoring devices StreamInsight Engine Standing Queries KPI Dashboards, SharePoint UI 1 Web servers Query Logic Query Logic 4 2 3 Trading stations Event stores & Databases Query Logic Event stores & Databases Stock ticker, news feeds

  21. Microsoft StreamInsight Event Event Event Event Event Event Event Event Event CEP Application Development Development experience with .NET, C#, LINQ and Visual Studio 2008/10 CEP platform from Microsoft to build event-driven applications Event targets Event sources Event-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementally CEP Engine Standing Queries Output Adapters Input Adapters Flexible adapter SDK with high performance to connect to different event sources and sinks The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data Static reference data

  22. Central Concepts • Events • User Payload + temporal properties • Type based on .NET structure or class • Point / Interval / Edge • Streams • Sequence of events with specific type • Flow into standing queries • Queries • Composed of query operators • Apply desired semantics on events • In-memory, event-triggered, incremental computations • Adapters • Produce / Consume event streams (Connection Manager)

  23. Typical Query & Analytics Patterns • Introduce additional event fields • Filter out events • Grouping by one or more event properties • Aggregation for each event group over a set (a window) • Rank events within windows • Correlate and synchronize event streams • Check for absence of events • Create result only when input changes • Remove duplicates • Reduce event rate, resample stream • Enrich events with reference data • Collection of assets may change over time

  24. LINQ Query Examples Filter Project &Aggregate Project Window LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; Grouping Join LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i intoSubStream fromwin inSubStream.HoppingWindow( FiveMinutes,ThreeSeconds) selectnew { i = SubStream.Key, a = win.Avg(e => e.f) };

  25. Extensibility • Need for domain-specific extensions • Integrate with functionality available in existing libraries • User-defined operators, functions, aggregates • Code written in .NET, deployed as .NET assembly • Window-based semantics for UDOs, UDAs • Receive a set of events • Produce a value / set of events

  26. Demo • Adapter model • Windows • Grouping • Altering event lifetime • Extending

  27. For More Information • Download:http://go.microsoft.com/fwlink/?LinkId=160598 • StreamInsight MSDN documentation:http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx • Blogs:http://blogs.msdn.com/streaminsight/ http://www.SQLIS.com/ • Forum:http://social.msdn.microsoft.com/Forums/en-US/streaminsight • StreamInsight MSDN portal:http://msdn.microsoft.com/en-us/ee476990.aspx • Samples:http://streaminsight.codeplex.com • Twitter:http://twitter.com/streaminsight

More Related