230 likes | 378 Views
Introduction to Message Passing. CMSC 34000 Lecture 3 1/11/05. Class goals. Parallel hardware and software issues First look at actual algorithm (trapezoidal rule for numerical integration) Introduction to message passing in MPI Networks and the cost of communication.
E N D
Introduction to Message Passing CMSC 34000 Lecture 3 1/11/05
Class goals • Parallel hardware and software issues • First look at actual algorithm (trapezoidal rule for numerical integration) • Introduction to message passing in MPI • Networks and the cost of communication
Parallel Hardware (Flynn’s Taxonomy) • SISD • MIMD • SIMD • MISD S = Single, M = Multiple I = Instruction stream D = Data stream
CPU (control and arithmetic) Memory (main & register Data/instruction transfer bottleneck Pipelining (multiple instructions operating simultaneously) Vectorizing (single instruction acts on vector register Cache -- memory hierarchy Von Neumann & Modern
Single CPU for control Many (scalar) ALUs with registers One clock Many CPUs Each has control and ALU Memory may be “shared” or “distributed” Synchronized? SIMD / MIMD
Shared memory MIMD • Bus (contention) • Switch (expensive) • Cache-coherency?
Distributed memory MIMD • General interconnection network (e.g. CS Linux system) • Each processor has its own memory • To share information, processors must pass (and receive) messages that go over the network. • Topology is very important
Different mesh topologies • Totally connected • Linear array/ring • Hypercube • Mesh/Torus • Tree / hypertree • Ethernet… • And others
Issues to consider • Routing (shortest path = best-case cost of single message) • Contention - multiple messages between different processors must share a wire • Programming: would like libraries that hide all this (somehow)
Numerical integration • Approximate • Using quadrature: • Repeated subdivision:
Finite differences • On (0,1): • At endpoints
System of Equations • Algebraic system of equations at each point • “Nearest neighbor stencil” • A row of the matrix looks like
Parallel strategy: Integration • Divide [a,b] into p intervals • Approximation on each subinterval • Sum approximations over each processor • How do we communicate? • Broadcast / reduction
Parallel strategy: Finite differences • How do we multiply the matrix by a vector (needed in Krylov subspace methods)? • Each processor owns: • A range of points • A range of matrix rows • A range of vector entries • To multiply by a vector (linear array) • Share the values at endpoints with neighbors
SPMD (Integration) • Single program running on multiple data • Summation over intervals • Particular points are different • Instances of program can talk to each other • All must share information at the same time • Synchronization
MPI • Message Passing Interface • Developed in 1990’s • Standard for • Sharing message • Collective communication • Logical topologies • Etc
Integration in MPI • Python bindings developed by Pat Miller (LLNL) • Ignore data types, memory size for now • Look at sample code
explicit send and receive O(p) communication cost (at best) Broadcast sends to all processes Reduce collects information to a single process Run-time depends on topology, implementation Two versions
Fundamental model of a message • Processor p “sends” • Processor q “receives” • Information needed: • Address (to read from/write to) • Amount of data being sent • Type of data • Tag to screen the messages • How much data actually received?
MPI Fundamentals • MPI_COMM_WORLD • MPI_Init() // import mpi • MPI_Comm_size() // mpi.size • MPI_Comm_rank() // mpi.rank • MPI_Finalize() // N/A
send receive non-blocking versions broadcast reduce other collective ops MPI Fundamentals
Communication costs over a network • Send, broadcast, reduce • Linear array • Point-to-point • Binary tree
Getting started with MPI on CS machines • Machines available (Debian unstable) • bombadil, clark, guts, garfield • mpich (installed on our system already) • mpicc is the compiler (mpic++, mpif77,etc) • mpirun -np x -machinefile hostsexecutableargs) • download pyMPI & build in home directory • http://sourceforge.net/projects/pympi/ • ./configure --prefix = /home/<you> • builds out of box (fingers crossed)