Manycore and the Microsoft .NET Framework 4 with Microsoft Visual Studio 2010

1. Required Slide SESSION CODE: DEV314 Manycoreand the Microsoft .NET Framework 4 with Microsoft Visual Studio 2010 Huseyin YILDIZ Software Design Engineer Microsoft Corporation

2. AGENDA • The need for parallelism • Overview of Parallel Extensions to the .NET Framework 4 • Task Parallel Library • Parallel LINQ • Concurrent collections and synchronization primitives

3. THE NEED FOR PARALLELISM • Manycore shift no longer news, but already fact of life • Moore’s law still holds, but increased transistors count goes to new cores • Quad-core is already the standard, 8, 12 and 16 cores becoming more common • Parallelism necessary in order to utilize new hardware • Parallelize CPU-bound work where applicable • Offload UI work for more responsive apps • Process more data or traverse more alternatives

4. HOW PARALLELISM HELPS • Things to consider • Efficiency • Load balancing • Scaling • Decomposability is key • Isolation • No mutable state CPU1 Sequential Op1 Op2 Op5 Op3 Op4 CPU1 Parallel CPU2 CPU3 CPU4 Time

5. PARALLELISM FROM GROUND UP? • Multithreading is hard, hand coded parallelism is harder • Thread management • Scalable scheduling algorithm and load balancing • Synchronization (task waits) • Asynchronous exception handling • Many pitfalls: • Races, deadlocks, lock convoys, memory model / cache coherency concerns, priority inversions… • High entry costs at best, poor scaling or instability at worst

6. LET’S PARALLELIZE THIS CODE… IEnumerable<Record> records = GetRecords(); List<Result> results = newList<Result>(); foreach(Recordrecord in records) { Result res = ProcessRecord(record); // non-trivial computation per record if (Filter(res)) results.Add(res); // check if result matches our criteria, if so add it into results } // can also be expressed as a simple LINQ statement List<Result> results = records.Select(rec => ProcessRecord(rec)).Where(res => Filter(res)).ToList();

7. HAND CODED PARALLELISM EXAMPLE Optimal concurrency? IEnumerable<Record> records = GetRecords(); List<Result> results = newList<Result>(); intnumWorkers = Environment.ProcessorCount; intactiveWorkers = numWorkers; IEnumerator<Record> enumerator = records.GetEnumerator(); ManualResetEventcompletionEvent = newManualResetEvent(false); for(int i = 0; i < numWorkers; i++) { ThreadPool.QueueUserWorkItem(delegate { // launch P workers while(true) { RecordcurrentRecord; lock (enumerator) { // grab a record under the lock before processing it if (!enumerator.MoveNext()) break; currentRecord = enumerator.Current; } Result res = ProcessRecord(currentRecord); // non-trivial computation per record if (Filter(res)) // check if result matches our criteria { lock(results) results.Add(res); // if so add it into results list under a lock } } if (Interlocked.Decrement(refactiveWorkers) == 0) completionEvent.Set(); }); } completionEvent.WaitOne(); // wait until all workers are done Must manage event lifetime Scalability bottleneck What if this throws? Scalability bottleneck Easy to introduce races Caller thread not utilized

8. PARALLEL EXTENSIONS TO THE RESCUE! • New parallelism APIs in .NET 4 • Parallel loops • Parallel LINQ • Fine grained (task-based) parallelism • New synchronization primitives and concurrent collections • Thread Pool enhancements • Goals • Allow developers to focus on application features instead of concurrency • Simple and expressive way of writing concurrent code • Performance and scaling on multicore hardware

9. SAME CODE PARALLELIZED USING PLINQ // Parallelized LINQ query List<Result> results = records.AsParallel().Select(rec => ProcessRecord(rec)).Where(res => Filter(res)).ToList();

10. QUICK CODE EXAMPLES FOR PARALLEL EXTENSIONS APIs Sequential Parallel for (int i = 0; i < 100; i++) DoWork(i); foreach (Record rec insrc) DoWork(rec); Parallel.For(0, 100, i => DoWork(i)); Parallel.ForEach(src, rec => DoWork(rec)); Record[] results = source .Select(rec => TransformFunc(rec)) .ToArray(); Record[] results = source.AsParallel() .Select(rec => TransformFunc(rec)) .ToArray(); { // independent statements A(); B(); C(); } Parallel.Invoke( () => A(), () => B(), () => C());

11. PARALLELISM SUPPORT IN .NET 4 AND VISUAL STUDIO 10 Tools Managed Concurrency Runtime and Programming Models Visual Studio IDE .NET Framework 4 Parallel LINQ Parallel Debugger Task Parallel Library Concurrent Collections Task Scheduler Concurrency Visualizer Sync Primitives Thread Pool CLR

12. PARALLEL LOOPS • Iterations must be independent • No guarantees for execution order or interleaving • Load balancing handled by runtime Parallel.ForEach(src, rec => DoWork(rec)); Parallel.For(0, 100, i => DoWork(i)); i=2 i=1 rec 2P+1 rec P+1 Thread 1 Thread 1 i=P*C+1 i=P*C+2 i=P*C+1 i=0 rec 1 src Thread 2 i=C+1 i=C+2 i=C+3 rec P rec 3P rec 2P Thread P Thread P

13. PARALLEL LOOPS (cont’d) • Synchronous from the caller’s POV • All iterations finish before returning, or AggregateException thrown • Common features • Degree of Parallelism control, Cancellation, Scheduler (ParallelOptions) • Loop Stop() / Break() • Thread local initializer / finalizers • Custom partitioners • Parallel.Invoke • Also supports ParallelOptions

14. PARALLEL LOOP CODE EXAMPLES // Breaking from a parallel loop Parallel.For(0, 1000, (i, state) => { if (SearchFunc(i)) state.Stop(); } // Controlling Degree Of Parallelism ParallelOptionspo = newParallelOptions() { MaxDegreeOfParallelism = 4}; Parallel.ForEach(source, po, element => ProcessRecord(element) } // Using thread local state Dictionary<Guid, int> counts = GetGlobalCounts(); Parallel.ForEach(enumSource, () => { //once per worker thread local init returnnewList<Result>(); }, (record, loopState, threadLocal) => { // actual loop body var result = ProcessRecord(record); if (result.InterestingFlag)threadLocal.Add(result); //cache results }, (threadLocal) => { //end of loop, once per worker combine delegate lock (counts) foreach (Result res inthreadLocal) counts[res.GUID] += res.Count; });

15. Parallel Loops in Action Basic loop functionality Use of thread locals in loops DEMO

16. PARALLEL LINQ • Parallel execution of LINQ to Objects queries • Easy to apply to existing LINQ based apps • Supports all standard LINQ operators • Only need to modify query declarations – iteration code remains same • Parallelism details hidden from the user • Works for any IEnumerable<T>

17. HOW PLINQ WORKS • Input data is partitioned into P disjoint sets • Operators replicated across partitions, and each runs on a separate thread • Results aggregated / merged • Example Query: (from x in src.AsParallel() where f(x)select y(x)).Sum(); Thread 1 Thread P where f(x) where f(x) select y(x) select y(x) Sum() Sum() src Sum()

18. PARALLEL LINQ (cont’d) • Exception handling and cancellation similar to Parallel.For/ForEach • Iterator throws AggregateException / OperationCanceledException • Operators for PLINQ-specific control: • WithDegreeOfParallelism(), WithMergeOptions(), WithExecutionMode() • User provided operator delegates must be independent • Quick note on scaling • Work per element • Element count

19. PLINQ Demo DEMO

20. TASK PARALLEL LIBRARY • Main API concepts • Task • Unit of asynchronous work • Waitable, cancelable, supports asynchronous exception handling • Also building block for parallel loops and PLINQ • Task<TResult> • A “waitable “result which is computed asynchronously • Continuations • TaskSchedulers

21. TASK PARALLEL LIBRARY – CREATING TASKS • Create & queue: Task.Factory.StartNew() • Delay started tasks: Task constructor + Task.Start() • Continuations • Task.ContinueWith() • Task.Factory.ContinueWhenAny() • Task.Factory.ContinueWhenAll() • Support for Aynchronous Programming Model • Task.Factory.FromAsync()

22. TASK WAITING, EXCEPTIONS AND CANCELLATION • Single and multiple wait APIs • Wait() • Task.WaitAll(), Task.WaitAny() • Exception handling similar to parallel loops and PLINQ • Wait() methods throw AggregateException • Also accesible from Task.Exception property • Unhandled Task exceptions • Cancellation

23. Task API Demo Basic task operation Continuations TPL Tasks and ThreadPool DEMO

24. THREADPOOL SCHEDULER IN .NET 3.5 Thread 2 Dispatch Loop • Single global queue • Protected by a shared lock Thread N Dispatch Loop Thread 1 Dispatch Loop Enqueue Dequeue T1 T3 T2 Global Queue (FIFO)

25. NEW THREADPOOL SCHEDULER IN .NET 4 Thread 2 Dispatch Loop • Shared by ThreadPool and TPL • Local work queues and work stealing (TPL only) • Improved thread management Thread N Dispatch Loop Thread 1 Dispatch Loop Enqueue Dequeue T1 T3 T2 T4 Global Queue (FIFO) Dequeue Enqueue T5 T6 T7 T8 Steal Steal Steal Thread 1 Local Queue (LIFO) Thread 2 Local Queue (LIFO) Thread N Local Queue (LIFO)

26. COORDINATION DATA STRUCTURES • Thread-safe, scalable collections • IProducerConsumerCollection<T> • ConcurrentQueue<T> • ConcurrentStack<T> • ConcurrentBag<T> • ConcurrentDictionary<TKey,TVal> • BlockingCollection<T> • Partitioning • {Orderable}Partitioner<T> • Partitioner.Create • Cancellation • CancellationTokenSource • CancellationToken • Synchronization • ManualResetEventSlim • SemaphoreSlim • SpinLock • SpinWait • Barrier • CountdownEvent • Initialization • Lazy<T> • LazyInitializer.EnsureInitialized<T> • ThreadLocal<T>

27. BLOCKING COLLECTION EXAMPLE // Producer consumer pattern BlockingCollection<string> bc = newBlockingCollection<string>(); // Start the producer Task Taskt1 = Task.Factory.StartNew(() => { while(!streamReader.EndOfStream) bc.Add(streamReader.ReadLine()); bc.CompleteAdding(); }); // Start the consumer Task Taskt2 = Task.Factory.StartNew(() => { try { // Consume from the blocking collection while(true) Console.WriteLine(bc.Take()); } catch(InvalidOperationException) { // IOE thrown from Take() indicated completed collection Console.WriteLine("That's All!"); } });

28. RECAP • Parallel Loops and PLINQ • Good starting point for speeding up existing apps • Apply at outer levels and CPU-bound stages first, • Task API for finer grained control • Also check out concurrency related tools in VS2010

29. Q&A

30. Required Slide Track PMs will supply the content for this slide, which will be inserted during the final scrub. Track Resources • ARC205: “Patterns of Parallel Programming” (Tue, 1:30 PM) • DEV408: “Task Parallel Library: Design Principles and Best Practices” (Wed 11:45 AM) • DEV317: “Profiling and Debugging Parallel Code with VS2010” (Thu 8:00 AM)

31. Required Slide Track PMs will supply the content for this slide, which will be inserted during the final scrub. Track Resources • Visual Studio – http://www.microsoft.com/visualstudio/en-us/ • Soma’s Blog – http://blogs.msdn.com/b/somasegar/ • MSDN Data Developer Center – http://msdn.com/data • ADO.NET Team Blog – http://blogs.msdn.com/adonet • WCF Data Services Team Blog – http://blogs.msdn.com/astoriateam • EF Design Blog – http://blogs.msdn.com/efdesign

32. Required Slide Resources Learning • Sessions On-Demand & Community • Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers • http://microsoft.com/technet • http://microsoft.com/msdn

33. Required Slide Complete an evaluation on CommNet and enter to win!

34. Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st http://northamerica.msteched.com/registration You can also register at the North America 2011 kiosk located at registrationJoin us in Atlanta next year

35. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Required Slide