1 / 44

Overview of Microsoft StreamInsight

Overview of Microsoft StreamInsight. Torsten Grabs Lead Program Manager Microsoft StreamInsight. The Need for an Event-Driven Platform. Event. Analytical results need to reflect important changes in business reality immediately and enable responses to them with minimal latency. request.

riona
Download Presentation

Overview of Microsoft StreamInsight

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of Microsoft StreamInsight Torsten Grabs Lead Program Manager Microsoft StreamInsight

  2. The Need for an Event-Driven Platform Event Analytical results need to reflect important changes in business reality immediately and enable responses to them with minimal latency request output stream input stream response

  3. Latency Scenarios for Event-Driven Applications Relational Database Applications CEP Target Scenarios Operational Analytics Applications, e.g., Logistics, etc. Data Warehousing Applications Web Analytics Applications Manufacturing Applications Financial trading Applications Monitoring Applications Aggregate Data Rate (Events/sec.)

  4. Example Scenarios • Manufacturing: • Sensor on plant floor • React through device controllers • Aggregated data • 10,000 events/sec • Power, Utilities: • Energy consumption • Outages • Smart grids • 100,000 events/sec • Web Analytics: • Click-stream data • Online customer behavior • Page layout • 100,000 events /sec • Financial Services: • Stock & news feeds • Algorithmic trading • Patterns over time • Super-low latency • 100,000 events /sec Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Data Stream Data Stream Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Stream Data Store & Archive Asset Specs & Parameters Event Processing Engine • Threshold queries • Event correlation from multiple sources • Pattern queries Lookup

  5. StreamInsight Platform StreamInsightApplication Development StreamInsight Application at Runtime Event sources Event targets Input Adapters Output Adapters StreamInsight Engine Devices, Sensors Pagers & Monitoring devices Standing Queries KPI Dashboards, SharePoint UI Web servers Query Logic Query Logic Trading stations Event stores & Databases Query Logic Event stores & Databases Stock ticker, news feeds

  6. What is Project “Austin”? • Real time data collection from wide variety of connected devices (Sensors, Smart Meters, Servers, Tablets, Phones) • Standards compliant endpoints (REST, XML, JSON) • Securable data ingress with data enrichment and transformation (geo-tagging, etc.) Connected • Multi-tenant Azure service with flexible, elastic capacity for collection and analytics • Federated scale out collection and analytics • Distributed service monitoring and tracing Scalable • Turn key connectivity for platform data sources and sinks (SQL Azure, Windows Azure Table Storage) • Integrated with Azure management portal and billing experiences Integrated • Rich temporal (StreamInsight) and sequential (Reactive Framework) analytics models • Dynamic, flexible query and data source management experience Analytics Cloud Services

  7. StreamInsight on Azure: “Austin” StreamInsightApplication Development StreamInsight Application at Runtime Prebuilt Input Adapters Prebuilt Output Adapters Austin StreamInsight Engine Standing Queries Data Egress Adapter RESTfulendpoint Stream-Insight Query Reactive Query Scalable Data Ingress Adapter Azure Tables Authentication Data Egress Adapter Built-in Archive Stream-Insight Query Management Service Monitoring Service

  8. Events • Events expose different temporal characteristics • Point in time events • Interval events with fixed duration • Interval events with initially unknown duration • Rich payloads capture all properties of an event d c e Payload/ value  b a t5 t3 t4 t1 t2 Time 

  9. Event Types • Events in Microsoft’s CEP platform use the .NET type system • Events are structured and can have multiple fields • Fields are typed using the .NET framework types • CEP engine provisioned timestamp fields capture all the different temporal event characteristics • Event sources populate time stamp fields

  10. Event Streams & Adapters • A stream is a possibly infinite sequence of events • Insertions of new events • Changes to event durations • Stream characteristics: • Event/data arrival patterns • Steady rate with end-of-stream indication • Intermittent, random, or in bursts • Out of order events: Order of arrival of events does not match the order of their application timestamps • Adapters • Receive/get events from the data source • Enqueue events for processing in the engine

  11. Typical CEP Queries • Typical CEP queries require combination of functionality • Complex type describes event properties • Calculations introduce additional event properties • Grouping by one or more event properties • Aggregation for each event group over a pre-defined period of time, typically a window • Multiple event groups monitored by the same query • Correlate event streams • Check for absence of activity with a data source • Enrich events with reference data • Collection of assets may change over time • We want to make writing and maintaining those queries easy or even effortless

  12. StreamInsight Query Features • Operators over streams • Calculations (PROJECT) • Correlation of streams from different data sources (JOIN) • Check for absence of activity with a data source (EXISTS) • Selection of events from streams (FILTER) • Stream partitioning (GROUP & APPLY) • Aggregation (SUM, COUNT, …) • Ranking and heavy hitters (TOP-K) • Temporal operations: hopping window, sliding window • Extensibility – to add new domain-specific operators

  13. LINQ Query Examples LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; Filter Project &Aggregate Project Window Grouping Join LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i intoSubStream fromwin inSubStream.HoppingWindow( FiveMinutes,ThreeSeconds) selectnew { i = SubStream.Key, a = win.Avg(e => e.f) };

  14. Extensibility SDK • Built-in operators do not cover all functionality • Need for domain-specific extensions • Integrate with functionality from existing libraries • Support for extensions in the CEP platform: • User-defined operators, functions, aggregates • Code written in .NET, deployed as .NET assembly • Query operators and LINQ can refer to functionality of the assembly • Temporal snap-shot operator framework • Interface to implement user-defined operators • Manages operator state and snapshot changes • Framework does the heavy lifting to deal with intricate temporal behavior such as out-of-order events

  15. Resiliency • Outages happen in computing • Power outages • “Patch Tuesday” • Human mistakes • Planned and unplanned downtime • Systems need to be “resilient” to outages • Minimize damage • Become operational again quickly • The specific requirements depend on how mission critical your applications is

  16. Resiliency: Timeliness • Timeliness: recover from outages quickly. • Goal is simple: as fast as possible. • StreamInsight doesn’t store event data, but it does store query state. • This may be significant. • This may be slow to recreate.

  17. Resiliency: Correctness Three Levels: • Exact equivalence. • The same stream of events, regardless of outage. • Equivalent events. • No missed events, and no wrong events, but duplicates are allowed. • Rough aggregation: get the moving average price of a stock over the last day. • Missing a few inputs will result in inaccurate, but close results. • Still don’t want to lose a day’s worth of work.

  18. What is Checkpointing? • Checkpointing saves a query’s state to disk. • You control when the checkpoint is initiated. • SI takes care of saving out consistent state. • After an outage, StreamInsight can restore this state. • This limits state loss during an outage, speeding recovery. • Level of correctness depends on additional work we are able to perform. • Recovery process is coordinated by SI.

  19. Checkpointing API public IAsyncResultserver.BeginCheckpoint( Queryquery, AsyncCallbackasyncCallback, object asyncState); public boolserver.EndCheckpoint( IAsyncResultasyncResult); public void server.CancelCheckpoint( IAsyncResultasyncResult);

  20. When is Checkpointing Useful? • Provides a mechanism to recover from an outage: • To recover from unexpected system failure. • To handle expected outages (e.g., patch Tuesday). • For machine migration. • Not a panacea: • Does not provide uninterrupted service. • Does not protect against broken query logic.

  21. Using Checkpoints • We’ll walk through the three progressively-strict checkpointing scenarios: • State retention. • Equivalent events. • Exact equivalence.

  22. Low Bar: State Retention • Ideal output: • Real output: … … C E D H’ H G G’ F F’ B B A A

  23. Checkpointing … … e e j j i i h h g g f f d d c c Enqueue markers into input streams to instruct operators to save their state.

  24. Checkpointing … e c d f g h i j oops c d f g h i j e …

  25. Recovery … … i i n n m m l l k k j j h h g g Load saved operator state and then start consuming input.

  26. Medium Bar: Equivalent Events • Ideal output: • Real output: … … C E D D H C G F B B B A A

  27. Filling the Gaps • StreamInsight needs help: • Missing state since last checkpoint. • Missed events during outage. • Solution: replayable adapters. • The dance: • StreamInsight picks a place in the input stream. • StreamInsight communicates this to the input adapter. • The input adapter replays from the chosen spot.

  28. Checkpointing … g f e c e d c e f e d f d g e h f h f g g i h g i j i j h h k i k j i l l j j k d e f g h i j k …

  29. Recovery … … g g l l k k j j i i h h f f e e

  30. A Place in the Stream … c e d h g f b a Physical Stream

  31. Communicating the State • Input adapter factories can optionally implement one of • IHighWaterMarkInputAdapterFactory • IHighWaterMarkTypedInputAdapterFactory • In a recovery situation, StreamInsight will then call Create with a high-water mark. • The factory is then responsible for properly cueing the input.

  32. StreamInsight in Action Internet of Things Demo

  33. The Demo Status Data Sensor Data StreamInsight “Austin” Alert Data Control Data Historical Data

  34. StreamInsight Design Principles • Scalability – Aggregate data rate keeps increasing. • Minimum resources impact (co-located). • Local computation • Avoid flooding the network • Programmability • Extensibility – UserDefinedAggregates, UserDefinedFunctions, UserDefinedOperators. • Composability. • Developer experience (language, IDE, debugging, supportability) • Adaptablity • Easy to integrate via adapters. • Portability (servers, edge devices)

  35. StreamInsight Architecture Host Process Runtime Engine Execution Operators Stream Manager Plan Manager Compiler Adap-ters Web Service Command Dispatcher Management Service Query Scheduler Synopsis Event Manager Diagnostics / Tracing Expression / Type Service Stream OS Metadata ...

  36. Management Service Highlights Host Process • Manageability API for query management (i.e. create, start, stop, delete query) and supportability / monitoring of running queries • Same manageability API for both embedded deployment and web service clients Runtime Engine Execution Operators Stream Manager Plan Manager Web Service Adapters Compiler Command Dispatcher Management Service Query Scheduler Synopsis Event Manager Metadata Diagnostics / Tracing Stream OS Expression / Type Service ...

  37. Compiler & Expressions Host Process Runtime Engine Execution Operators Stream Manager Plan Manager Web Service Compiler Adapters Command Dispatcher Highlights Management Service • Standardized IL allows us to implement a variety of syntactic surfaces over the algebra - e.g., LINQ, CQL, etc. • Allows for domain-specific front-end languages • Prepared for future extensions • Compile time type checking and type safe code generation for minimal runtime impact. • Support for UDF’s, UDAggs, UDOs. • JIT code generation for field references , expression evaluation for low latency processing of high event rates. • Basing on CLR helps leverage – • Code generator, JIT support • Type System • Tools and Libraries (LINQ Expressions, IDE, etc.) Query Scheduler Synopsis Event Manager Metadata Diagnostics / Tracing Stream OS Expression / Type Service ...

  38. Events & Streams Host Process Highlights • JIT code generation for field references, expression evaluation because interpreting these references is sub-optimal for low latency processing of high event rates. • Leverage JIT code generation support in CLR runtime for LINQ expressions. • Bind the query to different deployment environments based on the metadata. • Event manager is implemented as a combination of managed and native code in order to minimize overhead and ensure predictable performance. • Events are read-only and referenced-counted by streams (minimize data copying) Runtime Engine Execution Operators Stream Manager Plan Manager Compiler Web Service Adapters Command Dispatcher Management Service Query Scheduler Synopsis Event Manager Metadata Diagnostics / Tracing Stream OS Expression / Type Service ...

  39. Query Scheduler Host Process Runtime Engine Execution Operators Stream Manager Plan Manager Adapters Web Service Compiler Command Dispatcher . Management Service Query Scheduler Synopsis Event Manager Metadata Diagnostics / Tracing Stream OS Expression / Type Service ... Highlights • A query is executed by scheduling the individual operators as they become active. • Operator state transition is managed by the Scheduler. • When an operator becomes active a thread is scheduled for execution. • Scheduling decision based on priority of the query and other parameters. • Data flow architecture: reduced coupling and pipeline parallelism • Operators are affinitized to a thread/core (multi-core environments) to decrease lock contention and increase caching benefits. Periodic checks and migration for load balancing

  40. Execution Operators Host Process Runtime Engine Execution Operators Stream Manager Plan Manager Compiler Adapters Web Service Command Dispatcher Management Service XYZ Query Scheduler Synopsis Event Manager Stream OS Expression / Type Service Metadata Diagnostics / Tracing Union X,Y,Z ... XXX ZZZ YYY Highlights Apply Apply Apply • Efficient implementation of operators that perform incremental evaluation as each event is processed. • Clean, formal semantics. Leverage relational semantics whenever possible. • GroupAndApply Operator • Enables parallelism for scale-up (multi-core). • Groups are dynamically instantiated and torn down based upon the data. Large numbers of groups can be simultaneously active. (~50M active groups for MSN.com) AA BBB CCC Group A,B,C ABC

  41. The StreamInsight Team • Founded in 2008 based on incubation between MSR and SQL teams • Small team – by Microsoft standards  • Roles in Microsoft engineering teams • Program Managers: customer scenarios, functional specs, APIs, project mgmt, evangelism • Developers: architecture, technical design, product code, unit tests • Testers: test breakout, test code, lab runs, release signoff • Using agile development methods

  42. StreamInsightRoadmap StreamInsight2.1(on prem) StreamInsight on Azure (Cloud) StreamInsightservice on Windows Azure Currently private CTP GA this summer • Development experience • Major API overhaul Using Scrum to organize and manage schedules Work organized in sprints/milestones CTP (Community Technology Preview) after each milestone – similar to public beta TAP (Technology Adopter Program) as we get closer to the planned release

  43. For More Information • StreamInsight download location: http://go.microsoft.com/fwlink/?LinkId=160598 • StreamInsight blog: http://blogs.msdn.com/streaminsight/ • StreamInsight MSDN documentation: http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx • StreamInsight MSDN portal: http://msdn.microsoft.com/en-us/ee476990.aspx

More Related