A Deep Dive Into The Concurrency Runtime

1. TL22 A Deep Dive Into The Concurrency Runtime  NiklasGustafsson Software Architect Microsoft Corporation

2. Agenda • Introduction (10 min) • Scheduler / RM internals (15 min) • Scheduler / RM APIs (30 min)

3. Key Take-Aways • Why is Microsoft building this? • When is it available? • How do I build an algorithm library on the runtime? • How do I build a work scheduler on the runtime?

4. What you will not see… • Managed code • This is a C++ talk • Robots • This is not about CCR • Flashy demos • We’re going to be writing code • Impressive performance numbers • Your code will provide those later

5. Why A Concurrency Runtime? • The “CRT for concurrency” • Unleash the power of parallel hardware througha rich ecosystem of languages, libraries, and tools • No single company, algorithm, or scheduler is going to provide all concurrent solutions • Common infrastructure for partners and customers • Intel has announced integration with the concurrency runtime for TBB, OpenMP, and Parallel Studio

6. Resource Abundance • Different priorities • Efficiently maximizing throughput is primary concern, fairness is not (there’s enough to go around…) • Algorithm-driven scheduling • Minimize context switches • Manage data locality heuristically

7. Visual Studio 2010Concurrency platform and tools Tools Programming models PLINQ Parallel Pattern library Agents library Parallel Debugger Toolwindows Task Parallel library Data structures Data structures Concurrency runtime Profiler concurrency analysis Task scheduler Thread pool Task scheduler Resource manager Resource manager Operating system Threads Key: Managed library Native library Tools

8. Visual Studio 2010Concurrency platform and tools Tools Programming models PLINQ Parallel Pattern library Agents library Parallel Debugger Toolwindows Task Parallel library Data structures Data structures Concurrency runtime Profiler concurrency analysis Task scheduler Thread pool Task scheduler Resource manager Resource manager Operating system Threads Key: Managed library Native library Tools

9. Announced At Intel Developer Forum “A key challenge in parallelism is the management of the resources of the underlying hardware. In today’s announcement of Intel® Parallel Studio, we are also announcing that all the tools we’re building will work with the Microsoft® Concurrency Runtime, a common resource management and scheduling infrastructure, so that applications built using these tools will work seamlessly in the Windows environment.” — Renee James Vice President General Manager, Software and Services Group Intel Corporation

10. Concurrency RuntimeSupport for many programming models Programming models Parallel Pattern library Agents library … Intel Parallel Studio tools Threading building blocks OpenMP Concurrency runtime Task scheduler Resource manager Operating system Threads

11. Agenda • Introduction (10 min) • Scheduler / RM internals (25 min) • Scheduler / RM APIs (20 min)

12. Resource Manager • Serves requests for physical processors • With Windows XP / Windows Vista, it supports intra-process functionality • Further integration with Windows APIs (e.g. Windows Vista thread pool) in future releases • Basis for custom and third-party schedulers • For example, this is where Intel will hook in TBB

13. Resource Manager Example Scheduler A Scheduler B Resource manager Processor 0 Processor 1

14. Cooperative Task Scheduler • Similar to thread pool, scheduler is responsible for getting work items executed • Policy-driven • Basic scheduling primitives • Multiple scheduler instances

15. Scheduler Concepts • Virtual processor • Thread proxy, a worker thread • Context, a task dispatcher • Schedule group, a collection of queues • Work items • Chore, schedule via work-stealing queues • Light-weight task, scheduled via FIFO queues

16. Thread Pool In Action ThreadPool Queue Worker thread 1 Worker thread p … Item 4 Item 5 Item 1 Program thread Item 2 Item 3 Item 6

17. Work-Stealing In Action Work-stealing scheduler Local queue Local queue Global queue … Worker thread 1 Worker thread p … Task 6 Task 3 Task 4 Task 1 Program thread Task 5 Task 2

18. Scheduler Data Structures Context Context Context Pool of free contexts List of schedule groups Schedule group Schedule group Schedule group Schedule group Context Context Context Context Virtual processor Virtual processor Virtual processor Virtual processor

19. Schedule Group Structures Context Context Context LWT LWT LWT LWT Unblocked contexts FIFO tasks Chore Chore Chore Chore Chore Chore Chore Chore Per-context work-stealing queues

20. Dispatch Pseudo-Code • HRESULT Dispatch(IThreadProxy *pProxy) { • bool stop = false; • while( !stop ) { for each (group in groups) { boolfoundSomething = false; do { while (Task *tsk = group->GetUnblockedWork() != NULL) { tsk->Run(); foundSomething = true; } • while (Task *tsk = group->GetNewWork()) • { tsk->Run(); foundSomething = true; } • while (Chore *chre = group->Steal() != NULL) • { chre->Run(); foundSomething = true; } • } while (foundSomething); } stop = WaitForMoreWork(); } }

21. Agenda • Introduction (15 min) • Scheduler / RM internals (25 min) • Scheduler / RM APIs (20 min)

22. Scheduler APIs Parallel Pattern library Asynchronous agents Native task scheduler Resource manager

23. Important Scheduler APIs • Scheduler • Sets policy for a group or cores and tasks • SchedulerPolicy • Contains a set of policy elements for a scheduler instance • ScheduleGroup (in CTP: TaskGroup) • Supports tasks scheduled in FIFO queues • Context • Scheduler view of the worker thread • task_group, structured_task_group • Supports tasks scheduled in work-stealing queues • Event • Light-weight user-mode manual-reset event

24. Adding An Algorithm, Part 1 • Reduce: Turn vector or enumeration into single value • Examples • Sum, avg, min, max, product, exists

25. Adding An Algorithm, Part 2 • Alternator: Round-robin messaging block (agents)

26. Resource Manager Interface Parallel Pattern library Asynchronous agents Native task scheduler Resource manager

27. Important Resource Manager APIs • IResourceManager • Interface for the RM singleton, used to make resource requests and register schedulers • IScheduler • Represents a scheduler, each of which is allocated a set of resources • IVirtualProcessorRoot • Represents a processor, allocated to a scheduler • IThreadProxy • A worker thread, allocated by RM • IExecutionContext • A scheduler’s dispatcher

28. Scheduler / RM Startup Scheduler Resource manager CreateResourceManager IResourceManager::RequestInitialThreads IScheduler::GetPolicyValues IVirtualProcessor::Activate IExecutionContext::Dispatch

29. Writing A Simple Scheduler

30. Summary • Concurrency is a broad problem and requires broad solutions • No single programming model will be a silver bullet • The concurrency runtime provides a robust, efficient, foundations for building libraries for CPU-based parallelism • Partners and customers may “hook in” at many levels

31. PDC Parallelism Sessions • Microsoft Visual Studio: Bringing Out The Best In Multicore SystemsHazimShafiMonday, Oct. 27, 1:45 PM – 3:00 PM • Parallel Programming For C++ Developers In The Next Version Of Microsoft Visual StudioRick MolloyMonday, Oct. 27, 3:30 PM – 4:45 PM • Parallel Programming For Managed Developers With The Next Version Of Microsoft Visual StudioDaniel MothWednesday, Oct. 29, 10:30 AM – 11:45 AM • Concurrency Runtime Deep Dive: How To Harvest Multicore Computing ResourcesNiklasGustafssonWednesday, Oct. 29, 1:15 PM – 2:30 PM

32. PDC Parallelism Sessions • Addressing The Hard Problems Of Concurrency David Callahan, Lynne Hill Thursday, Oct. 30, 8:30 AM – 10:00 AM • Parallel Computing Application Architectures And Opportunities John Feo, Jerry Bautista (Intel)Thursday, Oct. 30, 10:15 AM – 11:45 AM • Future Of Parallel Computing (Panel) Speakers: Dave Detlefs, NiklasGustafsson, Sean Nordberg, James Reinders (Intel)Moderator: Selena Wilson Thursday, Oct. 30, 12:00 PM – 1:30 PM

33. Learn more about parallel computing at: MSDN.com/concurrency And download Parallel extensions to the .NET framework!

34. Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

35. Q&A Please use the microphones provided

36. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.