Multiprocessors and Multi-computers

1. Multiprocessors and Multi-computers • Multi-computers • Distributed address space accessible by local processors • Requires message passing • Programming tends to be more difficult • Multiprocessors • Single address space accessible by all processors • Simultaneous access to shared variables can produce inconsistent results • Generally programming is more convenient • Doesn’t scale to more than about sixteen processors

2. Memory Modules Bus Processes Memory modules Shared Memory Hardware Processes Bus configuration Crossbar switch configuration

3. Cache Coherence Significantly impacts performance • Cache Coherence Protocol • Write-Update: All caches immediately updated with altered data • Write-Invalidate: Altered data is invalidated in all caches. Updates take place only if subsequently referenced • False Sharing: Cache updates take place because multiple processes access the same cache block but not the same locations Memory y x y x Processor 2 Processor 1 Cache Blocks Note: Significant because each processor has a local cache

4. Shared Memory Access • Critical Section • A section of code that needs to be protected from simultaneous access • Mutual Exclusion • The mechanism used to enforce a critical section • Locks • Semaphores • Monitors • Condition Variables Shared Variable x =1 =2 Process 1 Process 2

5. Sequential Consistency Formally defined by Lamport (1979): • A multiprocessor result is sequentially consistent if: • The operations of each individual processors occur in proper sequence specified by its program. • The overall output matches some sequential order of operations by all the processors • Summary: Arbitrary interleaving of instructions does not affect the output generated.

6. Deadlock Resources permanently blocked waiting for needed resources • Necessary Conditions • Circular Wait • Limited Resource • Non-preemptive • Hold and Wait R1 R2 Rn-1 Rn P1 P2 Pn-1 Pn Deadly Embrace R1 R2 P1 P2 Two Process Deadlock

7. Locks Locks are the simplest mutual exclusion mechanism Normally, these are provided by operating system calls • Single bit variable: 1=locked, 0=unlocked “Enter door and lock the door at entry” • Spin locks (busy wait locks) • while (lock==1) spin(); // Normally involves hardware supportlock = 1;// Critical sectionlock = 0; • Advantages: Simple and easy to understand • Disadvantages • Poor use of the CPU if process does not block while waiting • It’s easy to skip the lock=0 statement • Examples: Pthreads and openMP provide OS abstractions Note: The while and lock setting must be atomic

8. Limits concurrent access An integer variable, s, controls the mechanism Operations P operation: passeren in Dutch for: to passs--;while (s<0) wait();// Critical section code V operation: vrigeven in Dutch for: to releases++;if (s<=0) unblock a waiting process; p(s); /* Critical section */ v(s); Notes Set s=1 initially for s to be a binary semaphore which acts like a lock. Set s=k>1 initially if k simultaneous entries are possible Set s=k<=0 for consumer processes waiting to consume data produced Disadvantage: Its easy to skip the v operation Example: UNIX OS Semaphores

9. Monitors • A Class mechanism that limits access to a shared resourcepublic class doIt{ public doIt() {//Constructor logic} public synchronized void critMethod() { wait(); // Wait till another thread signals notify();} } • Advantage: Most natural mutual exclusive mechanism • Disadvantage: Requires a language that supports the construct • Examples: Java, ADA, Modula II

10. Advantages: Reduce overhead with checking if a global variable reaches some value Avoids having to frequently “poll” the global variable Disadvantage: Its easy to skip the unlock operations Example: Pthreads Notes: wait() unlocks and locks mutex automatically Threads must already be waiting for a signal when it is thrown Example Thread 1 lock(mutex) while (c<>VALUE) wait(cVar,mutex) // Critical section unlock(mutex); Thread 2 if (c==VALUE) signal(condVar) Condition Variables Mechanism to guarantee a global condition before critical section entry

11. Shared Memory Programming Alternatives • Heavyweight processes • Modified syntax of an existing language (HP Fortran) • Programming language designed for parallel processing (ADA) • Compiler extensions to specify parallel execution (OpenMP) • Thread programming standard: Java Threads and pthreads

12. Threads Definition: Path of execution through a process • Heavyweight processes (UNIX fork, wait, waitpid, shmat, shmdt) • Disadvantage: time and memory expensive • Advantage: A blocked process doesn’t block the other processes • Lightweight threads (pthreads library) • Only needs to share stack space and instruction counter • "Thread Safe" programming required to guarantee consistent results • Pthreads • Threads can be spawned and started by other threads • They can run independently (detached from their parent thread) or require joins for termination • Formation of thread pools are possible • Threads communicate through signals • Processing order is indeterminate

13. Forks and Joins General thread flow of control pid = fork(); if (pid == 0) { /* Do spawned thread code */ } else { /* Do spawning thread code */ } if (pid == 0) exit(0); else wait(0); Note: Detached processes run independently from its parent without joins

14. Processes and Threads IP IP Code Heap Code Heap Stack Listeners Listeners Stack IP Resources Resources Stack Single Thread Process Dual Thread Process • Notes: • Threads can be three orders of magnitude faster than processes • Thread safe library routines can be used by multiple concurrent threads • Synchronization uses shared variables

15. Example Program (summing numbers) Heavyweight UNIX processes (Section 8.7.1) Pseudo code Create semaphores Allocate shared memory and attach shared memory Load array with numbers Fork child processes IF Parent THEN sum parent section ELSE sum child section P(semaphore) Add to global sum V(semaphore) IF (child) terminate ELSE join Print results Release semaphores, detatch and release shared memory Note: The Java and pthread version require about half the code

16. Modify Existing Language Syntax Example Constructs • Declaration of a shared memory variable shared int x; • Specify statements to execute concurrently par { s1(); s2(); s3(); … sn(); } • Iterations assigned to different processors forall (i=0; i<n; i++) { //code } • Examples: High Performance Fortran and C

17. Compiler Optimizations • The following works because the statements are independentforall (i = 0; i < P; i++) a[i] = 0; • Bernsteins conditions • Outputs from one processor cannot be inputs to another • Outputs from the processors cannot overlap • Example: a = x + y; b = x + z; are okay to execute simultaneously

18. Java Threads • Instantiate and run a thread ThreadClass t = new ThreadClass().start(); • Thread class Class ThreadClass extends Thread { public ThreadClass {//Constructor} public void run() { while (true) { //yield or sleep periodically. //thread code executed here. } } }

19. Pthreads IEEE POSIX 1003.1c 1995: UNIX-based C standardized API Advantages • Industry standardized interface which replaces vendor proprietary APIs • Thread creation, synchronization, and context switching are implemented in user space without kernel intervention, which is inherently more efficient than kernel-based thread operations • User-level implementation provides the flexibility to choose a scheduler that best suits the application, independent of the kernel scheduler. Drawbacks • Poor locality limits performance when accessing shared data across processors • The Pthreads scheduler hasn't proven suited to manage large numbers of threads • Shared memory multithreaded programs typically follow the SPMD model • Most parallel programs still are course-grain in design

20. Performance Comparisons Pthreads versus Kernel Threads Real: wall clock time (actual elapsed time) User: time spent in user mode Sys: time spent in the kernel within the process

21. Compiler Extensions (openMP) • Extensions for C/C++, Fortran, and Java (JOMP) • Consists of: Compiler directives, library routines and environment variables • Recognized industry standard developed in the late 1990s • Designed for shared memory programming • Uses fork-join model, but uses threads • Parallel sections of code execute “teams of threads” • General Syntax • C: #pragma omp <directive> • JOMP: //omp <directive>