ME964 High Performance Computing for Engineering Applications

ME964High Performance Computing for Engineering Applications Overview of MPI [Part 2 of 2] March 29, 2011 “There are three rules to follow when parallelizing large codes. Unfortunately, no one knows what these rules are.” W. Somerset Maugham and Gary Montry © Dan Negrut, 2011ME964 UW-Madison

Before We Get Started… • Last time • Start discussion of Message Passing Interface (MPI) standard • Mentioned the components that you need to install on your machine to run MPI code • Today • Learn how to run an MPI executable on Newton • Point-to-Point Communication with MPI • Collective Communication in MPI • Wrap up discussion of Message Passing Interface (MPI) standard • Other issues • Midterm Project: progress report due tonight • Syllabus has been updated, up-to-date info regarding deadlines, topics covered, etc.

Running on Local Machine mpiexec –n 4 MPI_PI.exe 13 (run this in the directory where you have your executable) number of processes command line argument (number of sub-intervals) Credit: Toby Heyn

Running on Newton (1 node) Credit: Toby Heyn

Running on Newton (2 nodes) Request exactly two nodes Credit: Toby Heyn

Running on Newton (2 nodes) Two cores per node Use exactly two nodes Credit: Toby Heyn

Running on Newton (2 nodes) Note 4 processes on two nodes (NEWTON04 and NEWTON06) Credit: Toby Heyn

Debugging MPI Code Under VS 2008, 32 Bit • Set MPIRun Command to “C:\Program Files\Microsoft HPC Pack 2008 R2\Bin\mpiexec.exe” (or wherever your mpiexec.exe is stored) • Set MPIShim Location to “C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\Remote Debugger\x86\mpishim.exe” (or wherever your mpishim.exe is stored) Step 2: Select MPI Cluster Debugger Step 1: Go to “Debugging”

Compiling MPI Code, Known Issue… • Why do I get a compilation error "catastrophic error: #error directive: SEEK_SET is #defined but must not be for the C++ binding of MPI" when I compile C++ application? • Define the MPICH_IGNORE_CXX_SEEK macro at compilation stage to avoid this issue. For instance, • mpicc-DMPICH_IGNORE_CXX_SEEK • Why? • There are name-space clashes between stdio.h and the MPI C++ binding. MPI standard requires SEEK_SET, SEEK_CUR, and SEEK_END names in the MPI namespace, but stdio.h defines them to integer values. To avoid this conflict make sure your application includes the mpi.h header file before stdio.h or iostream.h or undefine SEEK_SET, SEEK_CUR, and SEEK_END names before including mpi.h.

Outline • Introduction to message passing and MPI • Point-to-Point Communication • Collective Communication • MPI Closing Remarks

Example: Sending and Receiving[Point-To-Point Communication] #include<mpi.h> #include<stdio.h> int main(intargc, char **argv) { int i, my_rank, nprocs, x[4]; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &nprocs); if (my_rank == 0) { /* master */ x[0]=42; x[1]=43; x[2]=44; x[3]=45; for (i=1;i<nprocs;i++) MPI_Send(x, 4, MPI_INT, i, 0, MPI_COMM_WORLD); } else { /* worker */ MPI_Status status; MPI_Recv(x, 4, MPI_INT, 0, 0, MPI_COMM_WORLD, &status); if( my_rank==1 ) { printf("I am worker rank=%d/%d\n", my_rank,nprocs-1); printf("Here's the data I recently received: %d, %d, %d, %d", x[0], x[1], x[2], x[3]); } } MPI_Finalize(); exit(0); } destination and source user-defined tag Max number of elements to receive Useful when examined via calls like MPI_Get_count(), etc. Contains material from Allan Snavely’s presentation

Program Output

Process 0 Process 1 Send(data) Receive(data) MPI Basic Send/Receive(Point-to-Point Communication) • We need to fill in the details in • Things that need specifying: • How will “data” be described? • How will processes be identified? • How will the receiver recognize/screen messages? • What will it mean for these operations to complete? Credit: Rusty Lusk

M M P P Point-to-Point Communication • Data to be communicated is described by three attributes: • address • data type of the message • length of the message • Involved processes are described by two attributes • communicator • rank • A message is identified by a user chosen “tag” (integer) Credit: Allan Snavely

1 0 2 0 1 User-created Communicator 3 4 5 7 8 6 MPI Communicators User-created Communicator (default communicator) MPI_COMM_WORLD 4 0 1 2 3 9 5 6 7 8 14 10 11 12 13 19 15 16 17 18 Credit: Allan Snavely

MPI Tags • Messages are sent with an accompanying user-defined integer tag, to assist the receiving process in identifying the message • Messages can be screened at the receiving end by means of a specific tag, or not screened by using MPI_ANY_TAG as the tag in a receive • Some non-MPI message-passing systems have called tags “message types”. MPI calls them tags to avoid confusion with data types. Credit: Rusty Lusk

Tags and Contexts • Separation of messages used to be accomplished by use of tags, but • this requires libraries to be aware of tags used by other libraries. • this can be defeated by use of “wild card” tags. • Contexts are different from tags • no wild cards allowed • allocated dynamically by the system when a library sets up a communicator for its own use. • User-defined tags still provided in MPI for user convenience in organizing application • MPI_Comm_split can be used to create new communicators Credit: Rusty Lusk

MPI Point-to-Point Communication • Two modes of communication: • Blocking: Communication does not complete until the message has been received • Non-blocking: Completes as soon as the message is “on its way”, and hopefully it gets to destination • MPI provides four versions for these two modes • synchronous, buffered, standard, ready Contains material from Allan Snavely’s presentation

Synchronous/Buffered Sending in MPI • Synchronous with MPI_Ssend • In synchronous mode, a send will not complete until a matching receive is posted. • Copy data to the network, wait for an acknowledgement • The sender has to wait for a receive to be posted • No buffering of data • Buffered with MPI_Bsend • The send completes once the message has been buffered internally by MPI • Buffering incurs an extra memory copy • Does not require a matching receive to be posted • May cause buffer overflow if many bsends and no matching receives have been posted yet Contains material from Allan Snavely’s presentation

Standard/Ready Send • Standard with MPI_Send • Up to MPI to decide whether to do synchronous or buffered, for performance reasons • The rationale is that a correct MPI program should not rely on buffering to ensure correct execution • Ready with MPI_Rsend • May be started only if the matching receive has been posted • Can be done efficiently on some systems as no hand-shaking is required Contains material from Allan Snavely’s presentation

MPI_RECV • There is only one MPI_Recv, which returns when the data has been received • Performs a blocking receive operation • C Function prototype: voidMPI_Recv(void* buf, int count, MPI_Datatypedatatype, int source, int tag, MPI_Commcomm, MPI_Status* status); • buf - the initial address of the receive buffer (choice) (OUT) • count - the number of elements to be received (integer) (IN) • datatype - the data type of each receive buffer element (handle) (IN) • source - the rank of the source task in comm or MPI_ANY_SOURCE (integer) (IN) • tag - the message tag or MPI_ANY_TAG (positive integer) (IN) • comm - the communicator (handle) (IN) • status - the status object (INOUT).

... MPI_Ssend() MPI_Recv() ... ... MPI_Buffer_attach() MPI_Bsend() MPI_Recv() ... ... MPI_Buffer_attach() MPI_Bsend() MPI_Recv() ... ... MPI_Ssend() MPI_Recv() ... ... MPI_Buffer_attach() MPI_Bsend() MPI_Recv() ... ... MPI_Ssend() MPI_Recv() ... Deadlock Issues • Deadlock situations: appear when due to a certain sequence of commands the execution hangs Process 0 Process 1 Deadlock No Deadlock No Deadlock Contains material from Allan Snavely’s presentation

Deadlocking, Another Example • Recall that MPI_Sendis either synchronous or buffered • Example, on a certain machine running MPICH v1.2.1: Deadlock Process 0 Process 1 ... MPI_Send() MPI_Recv() ... ... MPI_Send() MPI_Recv() ... Data size > 127999 bytes Data size < 128000 bytes No Deadlock Contains material from Allan Snavely’s presentation

Avoiding Deadlocking • Easy way to eliminate deadlock is to pair MPI_Ssend and MPI_Recv operations the right way: Process 0 Process 1 ... MPI_Ssend() MPI_Recv() ... ... MPI_Recv() MPI_Ssend() ... No Deadlock • Conclusion: understand how the implementation works and what its pitfalls/limitations are

The Buffered Send • This subroutine is a blocking buffered mode send operation • It is a local operation. It does not depend on the occurrence of a matching receive in order to complete • If a send operation is started and no matching receive is posted, the outgoing message is buffered to allow the send call to complete • Return from an MPI_Bsenddoes not guarantee the message was sent • Message may remain in the buffer until a matching receive is posted. MPI_Buffer_Detach() will block until all messages are received

The Buffered Send[Cntd.] • Make sure you have enough buffer space available. An error occurs if the message must be buffered and there is there is not enough buffer space. • The amount of buffer space needed to be safe depends on the expected peak of pending messages. The sum of the sizes of all of the pending messages at that point plus (MPI_BSEND_OVERHEAD*number_of_messages) should be sufficient. You attach a buffer to a process using the following function: intMPI_Buffer_attach(void* buffer, intsize); • This subroutine provides MPI a buffer in the user's memory which is used for buffering outgoing messages. This buffer is used only by messages sent in buffered mode, and only one buffer is attached to a task at any time. • MPI_Bsend adds overhead because it requires an extra memory-to-memory copy of the outgoing data.

Non-blocking Communications • So far only blocking communication have been discussed: • The call returns whenever its operation is complete (MPI_SSEND returns once the message has been received, MPI_BSEND returns once the message has been buffered, etc..) • MPI provides non-blocking communication: the call returns immediately and there is another call that can be used to check on completion. • Rationale: Non-blocking calls let the sender/receiver do something useful while waiting for completion of the operation (without playing with threads, etc.). Contains material from Allan Snavely’s presentation

Non-blocking Communication • MPI_Issend, MPI_Ibsend, MPI_Isend, MPI_Irsend, MPI_Irecv MPI_Requestrequest; MPI_Isend(&x,1,MPI_INT,dest,tag,communicator,&request); MPI_Irecv(&x,1,MPI_INT,src,tag,communicator,&request); • Functions to check on completion: MPI_Wait, MPI_Test, MPI_Waitany, MPI_Testany, MPI_Waitall, MPI_Testall, MPI_Waitsome, MPI_Testsome. MPI_Statusstatus; MPI_Wait(&request, &status) /* blocks */ MPI_Test(&request, &status) /* doesn’t block */ Contains material from Allan Snavely’s presentation

MPI Barrier • Synchronization of the calling processes • The MPI_Barrier call blocks until all of the processes belonging to certain communicator have placed the call • No data is exchanged ... MPI_Barrier(MPI_COMM_WORLD) ... Credit: Allan Snavely

Example: Non-blocking Point-To-Point Communication #include"mpi.h" #include<stdio.h> int main(intargc, char **argv) { intmy_rank, x; MPI_Status status; MPI_Request request; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD,&my_rank); if (my_rank == 0) { /* Process 0 */ x=42; MPI_Isend(&x,1,MPI_INT,1,0,MPI_COMM_WORLD,&request); //.. // Do some work here, but don't change x (since might not have been sent yet) } elseif (my_rank == 1) { /* Process 1 */ int y; // Do some work here, you don’t need x here MPI_Irecv(&x,1,MPI_INT,0,0,MPI_COMM_WORLD,&request); // Do some work here, don't count on x yet, might not be here yet //... // Now I need x, i have to have it MPI_Wait(&request, &status); y = x/6; printf("I am worker rank=%d. The value of y: %d\n", my_rank, y); } MPI_Finalize(); return 0; }

Collective Communication • Operations that allow more than two processes to communicate simultaneously • MPI broadcast • MPI scatter • MPI gather • MPI reduce • All these can be built using point-to-point communications, but typical MPI implementations have optimized them, and it’s a good idea to use them • In all of these, all processes place the same call (in good SPMD fashion), although depending on the process, some arguments may not be used Contains material from Allan Snavely’s presentation

MPI Broadcast • One-to-many communication • Note that multicast can be implemented via the use of communicators (i.e., to create processor groups) ... MPI_Bcast(x, 4, MPI_INT, 0, MPI_COMM_WORLD) ... Rank of the root Credit: Allan Snavely

MPI Scatter • One-to-many communication • Not sending the same message to all (root) . . . (destinations) ... MPI_Scatter(x, 100, MPI_INT, y, 100, MPI_INT, 0, MPI_COMM_WORLD) ... Send buffer Receive buffer Rank of the root Data to send to each Data to receive Credit: Allan Snavely

MPI Gather • Many-to-one communication • Not sending the same message to the root . . . (sources) (root) ... MPI_Scatter(x, 100, MPI_INT, y, 100, MPI_INT, 0, MPI_COMM_WORLD) ... Send buffer Receive buffer Rank of the root Data to send from each Data to receive Credit: Allan Snavely

Gather-to-all • Many-to-many communication • Each process sends the same message to all • Different processes send different messages . . . . . . ... MPI_Allgather(x, 100, MPI_INT, y, 100, MPI_INT, MPI_COMM_WORLD) ... Send buffer Data to receive Receive buffer Data to send to each Credit: Allan Snavely

All-to-all • Many-to-many communication • Each process sends a different message to each other process . . . Block i from proc j goes to block j on proc i . . . ... MPI_Alltoall(x, 100, MPI_INT, y, 100, MPI_INT, MPI_COMM_WORLD) ... Send buffer Data to receive Receive buffer Data to send to each Credit: Allan Snavely

MPI Reduction Operations • Used to compute a result from data that is distributed across processes • Often what a user wants to do anyway - why not provide the functionality as a single API call rather than having people keep re-implementing the same things • Example: you have an large array and the first process stores the first 1000 entries, the second stores the next 1000, etc. • You are interested in the sum of all the elements in the array • Predefined operations: • MPI_MAX, MPI_MIN, MPI_SUM, etc. • It is possible to have user-defined operations

MPI_Reduce, MPI_Allreduce • MPI_Reduce: result is sent out to the root • The operation is applied element-wise for each element of the input arrays on each processor • MPI_Allreduce: result is sent out to everyone ... MPI_Reduce(x, r, 10, MPI_INT, MPI_MAX, 0, MPI_COMM_WORLD) ... input array output array array size root ... MPI_Allreduce(x, r, 10, MPI_INT, MPI_MAX, MPI_COMM_WORLD) ... Credit: Allan Snavely

MPI_Reduce example MPI_Reduce(sbuf,rbuf,6,MPI_INT,MPI_SUM,0,MPI_COMM_WORLD) sbuf P0 3 4 2 8 12 1 + rbuf P1 5 2 5 1 7 11 P0 11 16 20 22 24 18 + P2 2 4 4 10 4 5 + P3 1 6 9 3 1 1 11

MPI_Scan: Prefix reduction • Process i receives data reduced on process 0 to i. rbuf sbuf P0 P0 3 4 2 8 12 1 3 4 2 8 12 1 P1 P1 5 2 5 1 7 11 8 6 7 9 19 12 P2 P2 2 4 4 10 4 5 10 10 11 19 23 17 P3 P3 1 6 9 3 1 1 11 16 12 22 24 18 MPI_Scan(sbuf,rbuf,6,MPI_INT,MPI_SUM,MPI_COMM_WORLD) Credit: Allan Snavely

And more... • Most broadcast operations come with a version that allows for a stride (so that blocks do not need to be contiguous) • MPI_Gatherv(), MPI_Scatterv(), MPI_Allgatherv(), MPI_Alltoallv() • MPI_Reduce_scatter(): functionality equivalent to a reduce followed by a scatter • All the above have been created as they are common in scientific applications and save code • All details on the MPI Webpage Credit: Allan Snavely

MPI – We’re Scratching the Surface MPI_Abort, MPI_Accumulate, MPI_Add_error_class, MPI_Add_error_code, MPI_Add_error_string, MPI_Address, MPI_Allgather, MPI_Allgatherv, MPI_Alloc_mem, MPI_Allreduce, MPI_Alltoall, MPI_Alltoallv, MPI_Alltoallw, MPI_Attr_delete, MPI_Attr_get, MPI_Attr_put, MPI_Barrier, MPI_Bcast, MPI_Bsend, MPI_Bsend_init, MPI_Buffer_attach, MPI_Buffer_detach, MPI_Cancel, MPI_Cart_coords, MPI_Cart_create, MPI_Cart_get, MPI_Cart_map, MPI_Cart_rank, MPI_Cart_shift, MPI_Cart_sub, MPI_Cartdim_get, MPI_Comm_call_errhandler, MPI_Comm_compare, MPI_Comm_create, MPI_Comm_create_errhandler, MPI_Comm_create_keyval, MPI_Comm_delete_attr, MPI_Comm_dup, MPI_Comm_free, MPI_Comm_free_keyval, MPI_Comm_get_attr, MPI_Comm_get_errhandler, MPI_Comm_get_name, MPI_Comm_group, MPI_Comm_rank, MPI_Comm_remote_group, MPI_Comm_remote_size, MPI_Comm_set_attr, MPI_Comm_set_errhandler, MPI_Comm_set_name, MPI_Comm_size, MPI_Comm_split, MPI_Comm_test_inter, MPI_Dims_create, MPI_Errhandler_create, MPI_Errhandler_free, MPI_Errhandler_get, MPI_Errhandler_set, MPI_Error_class, MPI_Error_string, MPI_Exscan, MPI_File_call_errhandler, MPI_File_close, MPI_File_create_errhandler, MPI_File_delete, MPI_File_get_amode, MPI_File_get_atomicity, MPI_File_get_byte_offset, MPI_File_get_errhandler, MPI_File_get_group, MPI_File_get_info, MPI_File_get_position, MPI_File_get_position_shared, MPI_File_get_size, MPI_File_get_type_extent, MPI_File_get_view, MPI_File_iread, MPI_File_iread_at, MPI_File_iread_shared, MPI_File_iwrite, MPI_File_iwrite_at, MPI_File_iwrite_shared, MPI_File_open, MPI_File_preallocate, MPI_File_read, MPI_File_read_all, MPI_File_read_all_begin, MPI_File_read_all_end, MPI_File_read_at, MPI_File_read_at_all, MPI_File_read_at_all_begin, MPI_File_read_at_all_end, MPI_File_read_ordered, MPI_File_read_ordered_begin, MPI_File_read_ordered_end, MPI_File_read_shared, MPI_File_seek, MPI_File_seek_shared, MPI_File_set_atomicity, MPI_File_set_errhandler, MPI_File_set_info, MPI_File_set_size, MPI_File_set_view, MPI_File_sync, MPI_File_write, MPI_File_write_all, MPI_File_write_all_begin, MPI_File_write_all_end, MPI_File_write_at, MPI_File_write_at_all, MPI_File_write_at_all_begin, MPI_File_write_at_all_end, MPI_File_write_ordered, MPI_File_write_ordered_begin, MPI_File_write_ordered_end, MPI_File_write_shared, MPI_Finalize, MPI_Finalized, MPI_Free_mem, MPI_Gather, MPI_Gatherv, MPI_Get, MPI_Get_address, MPI_Get_count, MPI_Get_elements, MPI_Get_processor_name, MPI_Get_version, MPI_Graph_create, MPI_Graph_get, MPI_Graph_map, MPI_Graph_neighbors, MPI_Graph_neighbors_count, MPI_Graphdims_get, MPI_Grequest_complete, MPI_Grequest_start, MPI_Group_compare, MPI_Group_difference, MPI_Group_excl, MPI_Group_free, MPI_Group_incl, MPI_Group_intersection, MPI_Group_range_excl, MPI_Group_range_incl, MPI_Group_rank, MPI_Group_size, MPI_Group_translate_ranks, MPI_Group_union, MPI_Ibsend, MPI_Info_create, MPI_Info_delete, MPI_Info_dup, MPI_Info_free, MPI_Info_get, MPI_Info_get_nkeys, MPI_Info_get_nthkey, MPI_Info_get_valuelen, MPI_Info_set, MPI_Init, MPI_Init_thread, MPI_Initialized, MPI_Intercomm_create, MPI_Intercomm_merge, MPI_Iprobe, MPI_Irecv, MPI_Irsend, MPI_Is_thread_main, MPI_Isend, MPI_Issend, MPI_Keyval_create, MPI_Keyval_free, MPI_Op_create, MPI_Op_free, MPI_Pack, MPI_Pack_external, MPI_Pack_external_size, MPI_Pack_size, MPI_Pcontrol, MPI_Probe, MPI_Put, MPI_Query_thread, MPI_Recv, MPI_Recv_init, MPI_Reduce, MPI_Reduce_scatter, MPI_Register_datarep, MPI_Request_free, MPI_Request_get_status, MPI_Rsend, MPI_Rsend_init, MPI_Scan, MPI_Scatter, MPI_Scatterv, MPI_Send, MPI_Send_init, MPI_Sendrecv, MPI_Sendrecv_replace, MPI_Ssend, MPI_Ssend_init, MPI_Start, MPI_Startall, MPI_Status_set_cancelled, MPI_Status_set_elements, MPI_Test, MPI_Test_cancelled, MPI_Testall, MPI_Testany, MPI_Testsome, MPI_Topo_test, MPI_Type_commit, MPI_Type_contiguous, MPI_Type_create_darray, MPI_Type_create_f90_complex, MPI_Type_create_f90_integer, MPI_Type_create_f90_real, MPI_Type_create_hindexed, MPI_Type_create_hvector, MPI_Type_create_indexed_block, MPI_Type_create_keyval, MPI_Type_create_resized, MPI_Type_create_struct, MPI_Type_create_subarray, MPI_Type_delete_attr, MPI_Type_dup, MPI_Type_extent, MPI_Type_free, MPI_Type_free_keyval, MPI_Type_get_attr, MPI_Type_get_contents, MPI_Type_get_envelope, MPI_Type_get_extent, MPI_Type_get_name, MPI_Type_get_true_extent, MPI_Type_hindexed, MPI_Type_hvector, MPI_Type_indexed, MPI_Type_lb, MPI_Type_match_size, MPI_Type_set_attr, MPI_Type_set_name, MPI_Type_size, MPI_Type_struct, MPI_Type_ub, MPI_Type_vector, MPI_Unpack, MPI_Unpack_external, MPI_Wait, MPI_Waitall, MPI_Waitany, MPI_Waitsome, MPI_Win_call_errhandler, MPI_Win_complete, MPI_Win_create, MPI_Win_create_errhandler, MPI_Win_create_keyval, MPI_Win_delete_attr, MPI_Win_fence, MPI_Win_free, MPI_Win_free_keyval, MPI_Win_get_attr, MPI_Win_get_errhandler, MPI_Win_get_group, MPI_Win_get_name, MPI_Win_lock, MPI_Win_post, MPI_Win_set_attr, MPI_Win_set_errhandler, MPI_Win_set_name, MPI_Win_start, MPI_Win_test, MPI_Win_unlock, MPI_Win_wait, MPI_Wtick, MPI_Wtime • In some MPI implementations there are more than 300 MPI functions • Not all of them part of the MPI standard though, some vendor specific • Recall the 20/80 rule: six basic calls is probably what you need to implement a decent MPI code… • MPI_Init, MPI_Comm_Size, MPI_Comm_Rank, MPI_Send, MPI_Recv, MPI_Finalize

The PETSc Library • PETSc: Portable, Extensible Toolkit for Scientific Computation • One of the most successful libraries built on top of MPI • Intended for use in large-scale application projects, • Developed at Argonne National Lab (Barry Smith) • Open source, available for download at http://www.mcs.anl.gov/petsc/petsc-as/ • PETScprovides routines for the parallel solution of systems of equations that arise from the discretization of PDEs • Linear systems • Nonlinear systems • Time evolution • PETSc also provides routines for • Sparse matrix assembly • Distributed arrays • General scatter/gather (e.g., for unstructured grids)

PETSc PDE Numerical Solution Utilities ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Structure of PETSc

PETSc Numerical Components Nonlinear Solvers Time Steppers Newton-based Methods Other Euler Backward Euler Pseudo Time Stepping Other Line Search Trust Region Krylov Subspace Methods GMRES CG CGS Bi-CG-STAB TFQMR Richardson Chebychev Other Preconditioners Additive Schwartz Block Jacobi Jacobi ILU ICC LU (Sequential only) Others Matrices Compressed Sparse Row (AIJ) Blocked Compressed Sparse Row (BAIJ) Block Diagonal (BDIAG) Dense Matrix-free Other Distributed Arrays Index Sets Indices Block Indices Stride Other Vectors

Flow Control for PDE Solution Main Routine Timestepping Solvers (TS) Nonlinear Solvers (SNES) Linear Solvers (SLES) PETSc PC KSP Application Initialization Function Evaluation Jacobian Evaluation Post- Processing User code PETSc code

ME964 High Performance Computing for Engineering Applications