Asynchronous Execution

1. Asynchronous Execution

2. Benefits of asynchronous execution • Asynchronous execution techniques often desirable • share single processor more naturally across unrelated tasks • prioritize tasks relative to each other to ensure responsiveness • multiplex I/O operations to avoid cumulative waiting • perform computationally intensive tasks concurrently on multiple processors • Asynchronous execution is not always the right choice • task switching required to share processor is not free • timing related issues difficult to uncover or reproduce • successful test run != program correctness • must take explicit steps to protect shared data • not all libraries are prepared for concurrent access • Always test early and often on a real multi-processor system • fastest way to flush latent timing-related bugs out into the open

3. The asynchronous execution promise wall clock time Task A (ta) Task B (tb) Task C (tc) timetotal =(ta + tb + tc) Task A (ta) (sequential) Task B (tb) Task C (tc) timetotal =tb (parallel)

4. A B C A B C A B C B C Task switching overhead sharing a processor wall clock time original sequential timetotal = (ta + tb + tc) B multithreaded timetotal = (ta + tb + tc + task switching overhead )

5. Asynchronous execution options in the CLR • Asynchronous execution involves the use of threads • an independently scheduled path of execution through code • each thread has own callstack and copy of processor’s registers • scheduled in a prioritized, preemptive fashion for cpu time • The CLR provides a variety of multithreading techniques • most leverage a shared pool of threads managed by the CLR • may explicitly create and control a new thread • Every scenario provides opportunity for timing-related errors • starting threads is the easy part…

6. Using dedicated threads • System.Threading.Thread supports the creation and scheduling of separate threads • programmer controls thread creation • programmer controls thread priority • programmer can suspend/resume thread arbitrarily • programmer controls apartment affiliation • programmer controls thread name (aids in debugging) • Use-cases • performing very lengthy/blocking operations • prioritizing multiple tasks relative to each other • controlling COM apartment affiliation (interop scenarios only) • want thread’s existence to hold process running

7. Example: searching the Internet • Consider the case study below • UI is unresponsive for duration of the search • especially troublesome in GUI-based applications • no facility for displaying progress • no facility for allowing cancellation of search • search involves lots of blocking for network I/O class MyGoogle { static void Main( string[] args ) { Console.WriteLine("Searching the internet. Please wait."); string[] searchResults = SearchInternet(); ShowResults(searchResults); } static string[] SearchInternet() { // Do numerous lengthy blocking operations here... } }

8. T1 A single-threaded internet search Main SearchInternet UI unresponsive time ShowResults

9. Revision: multithreaded Internet search • A separate “search” thread allows application to maintain UI responsiveness to the interactive user • UI thread is not blocked during network I/O • UI thread can provide a progress indicator of some sort • UI thread can provide a means of stopping the search • search thread can be prioritized lower than UI thread

10. T1 T2 Goal: multithreaded internet search Main start search • maintain UI • check for cancellation • do other things... SearchInternet time ShowResults

11. Creating and starting a new thread • New threads are creating using the Thread class • encapsulates a single, independently scheduled thread • target method indicated using ThreadStart delegate • may be assigned a unique priority, name, apartment state • execution begins when Start method is called

12. Example: initiating search on a separate thread class MyGoogle { static string searchText; static string[] searchResults; static void Main( string[] args ) { searchText = args[0]; Thread t = new Thread(new ThreadStart(SearchInternet)); t.Priority = ThreadPriority.Lowest; t.Name = "Internet Search Thread"; t.Start(); Console.WriteLine("Searching the internet. Press ENTER to stop."); Console.ReadLine(); // Code to stop/wait here... (not yet shown) ShowResults(searchResults); } static void SearchInternet() { // Do numerous lengthy blocking operations here... } }

13. Stopping a thread • Execution ends when target method returns • as a result of method logic • as a result of an unhandled exception • Typical to set flag of some sort signaling other thread to exit • then call Join to wait until the CLR confirms thread has exited • May request the CLR more forcibly terminate other thread • by calling Thread.Interrupt or Thread.Abort • raises Thread{Interrupted,Abort}Exception in target • CLR makes best effort; not guaranteed

14. Example: stopping the search thread class MyGoogle { ... static volatile bool stopSearch = false; static void Main( string[] args ) { // Code to start thread not shown... Console.WriteLine("Searching the internet. Press ENTER to stop."); Console.ReadLine(); stopSearch = true; // Tell the search thread to exit. if( !t.Join(5000) ) { // Give the thread 5 secs to comply. t.Interrupt(); // Ask the CLR to stop the thread. t.Join(); // Wait for the CLR to do the job. } ShowResults(searchResults); } static void SearchInternet() { while( !stopSearch ) { // Do numerous lengthy blocking operations here... } } }

15. Example: putting it all together class MyGoogle { static string searchText; static string[] searchResults; static volatile bool stopSearch = false; static void Main( string[] args ) { searchText = args[0]; Thread t = new Thread(new ThreadStart(SearchInternet)); t.Priority = ThreadPriority.Lowest; t.Name = "Internet Search Thread"; t.Start(); Console.WriteLine("Searching the internet. Press ENTER to stop."); Console.ReadLine(); stopSearch = true; // Tell the search thread to exit. if( !t.Join(5000) ) { // Give the thread 5 secs to comply. t.Interrupt(); // Ask the CLR to stop the thread. t.Join(); // Wait for the CLR to do the job. } ShowResults(searchResults); } static void SearchInternet() { while( !stopSearch ) { // Do numerous lengthy blocking operations here... } } }

16. Thread pools • Sometimes the need for separate threads is more dynamic • number of required threads not known at design time • ex: server uses separate threads to handle incoming requests • ex: UI supports initiating unbounded set of “background” operations • Sometimes the cost of thread creation and startup dwarfs the duration of the task being executed • ex: using 4 threads to run 4 parts of a numeric calculation in parallel • Alternative is to “borrow” the use of pre-existing threads from a pool • pool initially contains small # of “ready” threads • pooled threads wait for work requests to be enqueued • requests take the form of a delegate identifying function to call • pool might dynamically add threads to meet increased demand • thread creation, coordination, and deletion managed by pool • cost of thread creation amortized over many small work requests

17. T T T T Thread pool usage model Work requests are queued to thread pool Pooled threads dequeue and invoke work requests Thread Pool ...do some things... queue a request for another thread to invoke a method on your behalf ...continue doing things... void SomeTargetMethod(){} Pooled thread invokesrequested method on yourbehalf

18. The CLR thread pool • The CLR provides one built-in thread pool for each process • Several mechanisms provide an interface onto that pool • Using Delegate.BeginInvoke • Using timers (multiple variations available) • System.Threading.ThreadPool provides niche features • Each share certain characteristics • pooled-threads offer scalability and ease of use • CLR creates threads based on various heuristics • thread count is capped to avoid saturation • cannot programmatically adjust/control heuristics • cannot cancel pending requests • cannot consistently prioritize threads relative to one another • a pooled thread’s existence will not hold a process running

19. Delegates • The most common form of asynchronous execution • compiler/CLR-synthesized methods support non-blocking method invocation • BeginInvoke queues request-to-call-delegate to CLR-managed thread pool • IAsyncResult returned for optional completion discovery • EndInvoke harvests target method return value and outputs (including exceptions thrown by target) • several ways to wait for completion and harvest results

20. The C# to CLR language mapping // C# public delegate double CalcPiDelegate( int precision ); // Psuedo C#/IL public sealed class CalcPiDelegate : System.MulticastDelegate { // Construction public .ctor( object target, unsigned native int methodToken ); // Invocation public double Invoke( int precision ); public IAsyncresult BeginInvoke( int precision, AsyncCallback cb, object state ); public double EndInvoke( IAsyncResult ar ); }

21. Example: calling BeginInvoke class App { static void Main() { CalcPiDelegate calcPi = new CalcPiDelegate(Math.CalculatePi); // Initiate the operation: calcPi.BeginInvoke(42, null, null); // Go do something else... } } class Math { public static double CalculatePi( int precision ) { double result = 3.1415927; if( precision > 7 ) { // A very long running calculation... } return(result); } }

22. T T T T Example: calling BeginInvoke Thread Pool static void Main(){ ...calcPi.BeginInvoke(42, null, null); // Do other things while PI // is being computed.} double CalculatePi( int precision ){ // Body omitted for brevity return(result);}

23. Harvesting results • Delegate.EndInvoke is used to harvest async call results • CLR reserves the right to leak if EndInvoke is never called • There are several ways to discover async call completion • poll • block for an arbitrary amount of time • request a callback upon completion (most common in UI apps) • Once operation completes, EndInvoke must be used to harvest return value and/or out parameters • EndInvoke can also be used to wait for completion with an implied infinite wait

24. Example: polling for completion class App { static void Main() { CalcPiDelegate calcPi = new CalcPiDelegate(Math.CalculatePi); // Initiate the operation: IAsyncResult ar = calcPi.BeginInvoke(42, null, null); Console.Write("Calculating PI..."); // Wait for the target method to be called: while( !ar.IsCompleted ) { Console.Write("."); Thread.Sleep(250); } // Harvest and process the results: double pi = calcPi.EndInvoke(ar); Console.WriteLine(pi); } }

25. Example: waiting without polling class App { static void Main() { CalcPiDelegate calcPi = new CalcPiDelegate(Math.CalculatePi); // Initiate the operation: IAsyncResult ar = calcPi.BeginInvoke(42, null, null); Console.Write("Calculating PI..."); // Wait no more than a quarter of a second for the target method // to be called and then process the result if didn’t timeout. if( ar.AsyncWaitHandle.WaitOne(250, true) ) { double pi = calcPi.EndInvoke(ar); Console.WriteLine(pi); } else { Console.WriteLine("operation timed out"); } } }

26. Example: callback-based completion notification using System.Runtime.Remoting.Messaging; // For AsyncResult class class App { static void Main() { CalcPiDelegate calcPi = new CalcPiDelegate(Math.CalculatePi); // Initiate the operation: calcPi.BeginInvoke(42, new AsyncCallback(OnPiComplete), null); Console.Write("Calculating PI..."); // Go do something else… } static void OnPiComplete( IAsyncResult iar ) { AsyncResult ar = (AsyncResult)iar; CalcPiDelegate calcPi = (CalcPiDelegate)ar.AsyncDelegate; double result = calcPi.EndInvoke(ar); Console.WriteLine(result); } }

27. T T T T Example: callback-based completion notification Thread Pool static void Main(){AsyncCallback cb = new AsyncCallback(OnPiComplete); calcPi.BeginInvoke(42, cb, null); // Do other things while PI // is being computed.} (1) double CalculatePi( int precision ){ // Body omitted for brevity return(result);} (2) void OnPiComplete( IAsyncResult ar ){ // Body omitted for brevity}

28. Dealing with concurrent execution • Reading data structures concurrently is not a problem • Updating data structures concurrently is a problem • e.g.: linked list temporarily disconnected during inserts/removals • two or more threads fighting over each others updates • readers seeing partially updated data while writer is updating • Some designs require other types of thread coordination • e.g.: producer/consumer designs • consumer thread(s) blocked if no data present • producer thread(s) signal presence of new data to consumer(s) • Thread synchronization is the act of explicitly coordinating thread execution to achieve desired behavior • the programmer and runtime are partners in this effort

29. Thread Synchronization in the CLR The CLR provides several thread synchronization tools • monitor • most common • integrated • others special purpose tools • Mutex class for deadlock-free multiple lock acquisition • events • multi-reader/single-writer support • interlocked integer update support

30. Wait-based synchronization • Most synchronization techniques require threads to make calls to the CLR that block until it is “safe” to proceed • programmer identifies resource(s) that need protection, or • programmer identifies other conditions governing execution • programmer codes calls to synchronization object as needed • CLR/framework blocks thread(s) until condition is achieved • Semantics of when it’s “safe” to proceed defined by the construct being used • one thread at a time, resource available, arbitrary condition… • Making the right calls is voluntarily • programmer(s) must carefully choreograph things • a thread that does not make the right calls can still access the resource being protected – but with undefined results

31. Monitor • The Monitor is the primary thread synchronization tool • each object has a demand-allocated “lock” called a SyncBlock • lock acquired/released using static Monitor.Enter/Exit • only one thread at a time allowed through Enter call • some languages provide exception-safe wrapper • Programmers agree on the object used for synchronization • this is the key! • “locking” doesn’t prevent access to objects by threads that don’t grab lock before accessing object • object being accessed is not required to be object used for synchronization

32. T1 T2 Using a monitor for synchronization safe to access data protected by this lock working (owns lock) working Enter Exit agreed upon object Enter Exit working (owns lock) working blocked while lock owned by another thread safe to access data protected by this lock

33. Example: using a Monitor public class Foo { public void ThreadSafeOperation() { Monitor.Enter(this); // Do something that requires single-threaded access Monitor.Exit(this); } public void JustAsThreadSafeOperation() { lock(this) { // Do something that requires single-threaded access } } } public void JustAsThreadSafeOperation() { Monitor.Enter(this); try { // Do something that requires... } finally { Monitor.Exit(this); } } Compiler translation

34. Example: protecting static fields public class Foo { private static int PrevValue = 0; private static int NextValue = 1; public static void ThreadSafeOperation() { lock( typeof(Foo) ) { PrevValue = NextValue; NextValue++; if( NextValue > 100 ) NextValue = 0; } } }

35. Summary • The CLR provides a variety of multithreading options • dedicated threads for long running, prioritized tasks • delegates for scalable, short-duration scenarios • All asynchronous execution techniques require the programmer to deal with thread synchronization • monitor & per-object locks provides the primary defense against dangerous concurrent access

36. For more information, please contact: Uladzimir Tsikhon Software Engineering Manager, Belarus Recourse Development Department EPAM Systems, Inc. Belarus, MinskPhone: +375(17) 2101662 ext 1756 Fax: +375(17) 2101168 Email: uladzimir_tsikhon@epam.com http://www.epam.com