Curing Your Event Processing Blues with Reactive Extensions (Rx)

1. Curing Your Event Processing Blues with Reactive Extensions (Rx) Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation

2. The Event Processing Landscape GPS RSS feeds Stock tickers Social media UI events Server management

3. Reactive Extensions Architecture from quote in stock wherequote.Symbol == “MSFT” selectquote.Value Windowing Grouping Sampling Filtering Recovery Sharing Projection Merging Throttling LINQ to Events Timeout Aggregating Joins IObserver<T> IObservable<T> IScheduler ISubject<T> Event Streams Time Cloud Threads Dispatchers Concurrency

4. Reactive Extensions Architecture from quote in stock wherequote.Symbol == “MSFT” selectquote.Value Windowing Grouping Sampling Filtering Recovery Sharing Projection Merging Throttling LINQ to Events Timeout Aggregating Joins IObserver<T> IObservable<T> IScheduler ISubject<T> Event Streams Time Cloud Threads Dispatchers Concurrency

5. Event Streams • Towards a unified programming model • Producers are observable sequences • .NET events, WinRT events, sensor APIs, APM methods, tasks, etc. • Consumers are observers • Hooking up “continuations” or handlers Observable Subscribe Observer

6. Essential Interfaces • namespace System • { • public interfaceIObservable<out T> • { • IDisposableSubscribe(IObserver<T> observer); • } • public interfaceIObserver<in T> • { • voidOnNext(T value); • voidOnError(Exception error); • voidOnCompleted(); • } • }

7. Notification Grammar OnNext(42) OnNext(43) OnCompleted source1 OnNext(“Hello”) OnError(error) source2 OnNext* (OnError | OnCompleted)?

8. Limitations of .NET Events Can’t pass around Hidden data source • exchange.StockTick += (sender, args) => • { • if (args.Quote.Symbol == “MSFT”) • { • // Imperative code • } • }; • exchange.StockTick -= /* what goes here? */; Lack of composition Hard resource maintenance

9. Observable Sequences to the Rescue Source of Quotes Objects can be passed • IObservable<Quote> stockQuotes = …; • varmsft= stockQuotes • .Where(quote => quote.Symbol == “MSFT”); • varsubscription = msft.Subscribe(quote => /* … */); • subscription.Dispose(); Can define query operators Easy resource maintenance

10. Are .NET Events Obsolete? • .NET Events • Code centric • Design-time experience • Not first class • Non-compositional • Lightweight • Rigid execution model (IL) • Observables • Data centric • No design-time experience • First class objects • Rich composition • Slightly more cost • Translatable with expression trees

11. demo Events versus Observables Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team

12. Reactive Extensions Architecture from quote in stock wherequote.Symbol == “MSFT” selectquote.Value Windowing Grouping Sampling Filtering Recovery Sharing Projection Merging Throttling LINQ to Events Timeout Aggregating Joins IObserver<T> IObservable<T> IScheduler ISubject<T> Event Streams Time Cloud Threads Dispatchers Concurrency

13. Dictionary Suggest Asynchronous request IObservable<string> “React” React Dictionary web service Reaction Reactive Reactor IObservable<DictionaryWord[]> Data bindingon UI thread

14. Converting Events and Asynchronous Methods • // Convert the TextChanged event to IObservable<string> • varinput = (from evtin Observable.FromEventPattern(txt, “TextChanged”) • select ((TextBox)evt.Sender).Text) • .Throttle(TimeSpan.FromSeconds(0.5)) • .DistinctUntilChanged(); • // Convert asynchronous proxy methods to Func<string, IObservable<string[]>> • varlookup = Observable.FromAsyncPattern<string, string[]>(svc.BeginLookup, • svc.EndLookup); • // Compose both sources using a query • varres = from term in input • from words in lookup(term).TakeUntil(input) • select words; No longer needed in Rx v2.0 for .NET 4.5, using Task<T>

15. demo Querying Event Streams Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team

16. Stock Trade Analysis MSFT 27.01 INTC 21.75 MSFT 27.96 MSFT 31.21 INTC 22.54 INTC 20.98 MSFT 30.73 ticks fromtickinticks

17. Stock Trade Analysis MSFT 27.01 INTC 21.75 MSFT 27.96 MSFT 31.21 INTC 22.54 INTC 20.98 MSFT 30.73 ticks 27.01 27.96 30.73 31.21 MSFT 21.75 22.54 20.98 INTC fromtickinticks grouptickbytick.Symbol

18. Stock Trade Analysis MSFT 27.01 INTC 21.75 MSFT 27.96 MSFT 31.21 INTC 22.54 INTC 20.98 MSFT 30.73 ticks [27.01, 27.96] [27.96, 31.21] [31.21, 30.73] MSFT [22.54, 20.98] [21.75, 22.54] INTC fromtickinticks grouptick bytick.Symbolintocompany fromopenCloseincompany.Buffer(2, 1)

19. Stock Trade Analysis MSFT 27.01 INTC 21.75 MSFT 27.96 MSFT 31.21 INTC 22.54 INTC 20.98 MSFT 30.73 ticks 0.034 0.104 -0.015 MSFT -0.069 0.036 INTC fromtickinticks grouptickbytick.Symbolintocompany fromopenCloseincompany.Buffer(2, 1) letdiff = (openClose[1] – openClose[0]) / openClose[0]

20. Stock Trade Analysis MSFT 27.01 INTC 21.75 MSFT 27.96 MSFT 31.21 INTC 22.54 INTC 20.98 MSFT 30.73 ticks 0.034 0.104 -0.015 MSFT -0.069 0.036 INTC fromtickinticks grouptickbytick.Symbolintocompany fromopenCloseincompany.Buffer(2, 1) letdiff = (openClose[1] – openClose[0]) / openClose[0] wherediff > 0.1

21. Stock Trade Analysis MSFT 27.01 INTC 21.75 MSFT 27.96 MSFT 31.21 INTC 22.54 INTC 20.98 MSFT 30.73 ticks res Company = MSFT Increase = 0.104 fromtickinticks grouptickbytick.Symbolintocompany fromopenCloseincompany.Buffer(2, 1) letdiff = (openClose[1] – openClose[0]) / openClose[0] wherediff > 0.1 selectnew{ Company = company.Key, Increase = diff }

22. demo Complex Event Processing Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team

23. Reactive Extensions Architecture from quote in stock wherequote.Symbol == “MSFT” selectquote.Value Windowing Grouping Sampling Filtering Recovery Sharing Projection Merging Throttling LINQ to Events Timeout Aggregating Joins IObserver<T> IObservable<T> IScheduler ISubject<T> Event Streams Time Cloud Threads Dispatchers Concurrency

24. The Role of SchedulersParameterize Concurrency // // Runs a timer on the default scheduler // IObservable<long> Timer(TimeSpandueTime); // // Every operator that introduces concurrency // has an overload with an IScheduler // IObservable<long> Timer(TimeSpandueTime, IScheduler scheduler);

25. The Role of SchedulersSynchronization varxs = Observable.Return(42, Scheduler.ThreadPool); xs.Subscribe(x => lbl.Text = "Answer = " + x); xs.ObserveOn( frm ) .Subscribe(x => lbl.Text = "Answer = " + x); xs.ObserveOn(new ControlScheduler(frm)) .Subscribe(x => lbl.Text = "Answer = " + x);

26. The IScheduler Interface • public interfaceIScheduler • { • DateTimeOffset Now { get; } • IDisposable Schedule<TState>( TState state, • Func<IScheduler, TState, IDisposable> action); • IDisposableSchedule<TState>( TimeSpandueTime,TState state, • Func<IScheduler, TState, IDisposable> action); • IDisposableSchedule<TState>( DateTimeOffsetdueTime,TState state, • Func<IScheduler, TState, IDisposable> action); • }

27. Operational Layering of Rx • publicstaticIObservable<T> Return<T>(T value, • IScheduler scheduler) • { • returnObservable.Create<T>(observer => • { • // Serialize state to scheduler; return ability to cancel • returnscheduler.Schedule(new { value, observer }, (_, x) => • { • x.observer.OnNext(x.value); • x.observer.OnCompleted(); • returnDisposable.Empty; // No recursive work • }); • }); • }

28. Virtualizing Time for Testing • varscheduler = newTestScheduler(); • varinput = scheduler.CreateHotObservable( • OnNext(300, “Bart De Smet”), • OnNext(400, “Erik Meijer”), • OnCompleted<string>(500) • ); • var results = scheduler.Start(() => from name ininput • selectname.Length); • results.Messages.AssertEqual( • OnNext(300, 12), • OnNext(400, 11), • OnCompleted<int>(500) • );

29. demo Testing in Virtual Time Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team

30. Expression Tree Representation of Queries • Runtime translation of reactive queries • Like IQueryable<T> for classic LINQ • Examples: • LINQ to WMI Events • Translates LINQ queries into WQL statements • LINQ to PowerShell • Translates LINQ queries into PowerShell pipelines • LINQ to Twitter • Translates LINQ queries into Twitter queries • Etc.

31. The IQbservable<T> Interface • public interfaceIQbservable<out T> : IObservable<T> • { • TypeElementType { get; } • ExpressionExpression { get; } • IQbservableProvider Provider { get; } • } • public interfaceIQbservableProvider • { • IQbservable<T> CreateQuery<T>(Expressionexpression); • }

32. The Query Processing Landscape IQueryable<T> IQbservable<T> • Expression trees • LINQ to SQL (T-SQL) • Expression trees • LINQ to WMI (WQL) Query language translation LINQ IEnumerable<T> IObservable<T> Local execution(in-process IL) • Iterators (yield) • LINQ to Objects • Observable.Create • LINQ to Events (Rx) Pull-based Push-based

33. demo LINQ to WMI Events Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team

34. announcing StreamInsight v2.1 Now with support for hosting Rx v1.0 queries using IQbservable<T>

35. The Future of RxSupport for .NET Framework 4.5 • Synergy with asynchronous programming features • Prefer using Task<T> for single-value asynchrony • Deprecation of FromAsyncPattern • Convert to IObservable<T> for more power • Use IObservable<T> for event streams • Await support returns last value • Reduce blocking operations • Deprecation of First, Last, Single, ForEach • Leveraging new platform capabilities • E.g. ExceptionDispatchInfo

36. The Asynchronous Programming Landscape IEnumerable<T> IObservable<T> var res = from s in stocks wheres.Symbol == “MSFT” selectq.Quote res.Subscribe(x => … var res = from p in products wherep.Name == “Rx” selectp.Price; foreach (var x in res) … Multiple values (*) Func<T> Task<T> Single value (1) var y = f(x); varz = g(y); var y = awaitfAsync(x); varz = awaitgAsync(y); Synchronous Asynchronous

37. Async without Rx… • asyncTask<string> GetHtmlAsync(Uri url) • { • var client = newWebClient(); • var download = client.DownloadStringAsync(url); • var timeout = Task.Delay(TimeSpan.FromSeconds(30)); • if (awaitTask.WhenAny(download, timeout) == timeout) • thrownewTimeoutException(); • var html = await download; • return html; • } Imperative glue~ WaitForMultipleObjectsEx

38. Async with Rx… Better together! • asyncTask<string> GetHtmlAsync(Uri url) • { • var client = newWebClient(); • var download = client.DownloadStringAsync(url) • .ToObservable() • .Timeout(TimeSpan.FromSeconds(30)); • varhtml = await download; • return html; • } Composition of query operators Await support for IObservable<T>

39. The Future of RxInteroperability with WinRT • Use WinRT-style events with FromEventPattern • Different method signatures in IL • New IScheduler implementations • For WinRTThreadPool and CoreDispatcher • ObserveOnDispatcher and SubscribeOnDispatcher support • Conversions of IAsync* types • Smooth interop with WinRT asynchronous operations • Support for progress tracking using IObservable<T>

40. The Future of RxTowards Portable Library Support System.Reactive.PlatformServices Platform Enlightenments System.Reactive.Providers Expression tree support System.Reactive. Windows.Threading System.Reactive. Windows.Forms System.Reactive. WindowsRuntime System.Reactive. Remoting System.Reactive.Linq Query operators System.Reactive.Core Base classes, core schedulers, extensions methods System.Reactive.Interfaces Stable interfaces for forward compatibility

41. The Future of RxPerformance • Reducing locking in subjects [3x] • Faster producers [100-1000x] • New ISchedulerLongRunning interface • Pipeline throughput [4x] • One call frame per operator • Reduced wrapping of observers • Less operators defined in terms of others • Eliminating redundant locks • Eradicating time skew due to timing issues • Play nicer with GC

42. demo Preview of Rx v2.0 Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team

43. Summary • Tame your event streams using Rx and LINQ! • Download Rx today! • Through http://www.microsoft.com/download (search for Rx SDK) • Using NuGet @ www.nuget.org (search for Rx-Main) • Now available: Rx v2.0 RC • Watch videos at http://channel9.msdn.com/tags/Rx • Visit our forums on MSDN

44. Related Content • DEV414 – LINQ, Take Two – Realizing the LINQ to Everything Dream Find Me Later This Week In The Ask The Experts Area

45. Resources Learning TechNet • Connect. Share. Discuss. • Microsoft Certification & Training Resources http://europe.msteched.com www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers • http://microsoft.com/technet http://microsoft.com/msdn

46. Evaluations Submit your evals online http://europe.msteched.com/sessions

47. © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.