1 / 17

PGAS Languages and Halo Updates

PGAS Languages and Halo Updates. Will Sawyer, CSCS. Important concepts and acronyms. PGAS: Partitioned Global Address Space UPC: Unified Parallel C CAF: Co-Array Fortran Titanium: PGAS Java dialect MPI: Message-Passing Interface SHMEM: Shared Memory API (SGI).

lewis
Download Presentation

PGAS Languages and Halo Updates

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PGAS Languages and Halo Updates Will Sawyer, CSCS

  2. Important concepts and acronyms PGAS: Partitioned Global Address Space UPC: Unified Parallel C CAF: Co-Array Fortran Titanium: PGAS Java dialect MPI: Message-Passing Interface SHMEM: Shared Memory API (SGI)

  3. Partitioned Global Address Space • Global address space: any thread/process may directly read/write data allocated by any other • Partitioned: data is designated as local (with ‘affinity’) or global (possibly far); programmer controls layout x: 1 y: x: 5 y: x: 7 y: 0 l: l: l: g: g: g: By default: Object heaps are shared Program stacks are private p0 p1 pn Global address space 3 Current languages: UPC, CAF, and Titanium POMPA Kickoff Meeting, May 3-4, 2011

  4. Potential strengths of a PGAS language • Interprocess communication intrinsic to language • Explicit support for distributed data structures (private and shared data) • Conceptually the parallel formulation can be more elegant • One-sided shared-memory communication • Values are either ‘put’ or ‘got’ from remote images • Support for bulk messages, synchronization • Could be implemented with message-passing library or through RDMA (remote direct memory access) • PGAS hardware support available • Cray Gemini (XE6) interconnect supports RDMA • Potential interoperability with existing C/Fortran/Java code POMPA Kickoff Meeting, May 3-4, 2011

  5. POP Halo Exchange with Co-Array Fortran • Worley, Levesque, The Performance Evolution of the Parallel Ocean Program on the Cray X1, Cray User Group Meeting, 2004 • Cray X1 had a single vector processor per node, internode comm. hardware support • Co-Array Fortran (CAF) driven by Numrich, et al., also the authors of SHMEM • Halo exchange programmed in MPI, CAF, SHMEM

  6. Halo Exchange “Stencil 2D” Benchmark • Halo exchange and stencil operation over a square domain distributed over a 2-D virtual process topology • Arbitrary halo ‘radius’ (number of halo cells in a given dimension, e.g. 3) • MPI implementations: • Trivial: post all 8 MPI_Isend and Irecv • Sendrecv: MPI_Sendrecv between PE pairs • Halo: MPI_Isend/Irecv between PE pairs • CAF implementations: • Trivial: simple copies to remote images • Put: reciprocal puts between image pairs • Get: reciprocal gets between image pairs • GetA: all images do inner region first, then • all do block region (fine grain, no sync.) • GetH: half of images do inner region first, • half do block region first (fine grain, no sync.)

  7. Example code: Trivial CAF real, allocatable, save :: V(:,:)[:,:] : allocate( V(1-halo:m+halo,1-halo:n+halo)[p,*] ) : WW = myP-1 ; if (WW<1) WW = p EE = myP+1 ; if (EE>p) EE = 1 SS = myQ-1 ; if (SS<1) SS = q NN = myQ+1 ; if (NN>q) NN = 1 : V(1:m,1:n) = dom(1:m,1:n) ! internal region V(1-halo:0, 1:n)[EE,myQ] = dom(m-halo+1:m,1:n) ! to East V(m+1:m+halo, 1:n)[WW,myQ] = dom(1:halo,1:n) ! to West V(1:m,1-halo:0)[myP,NN] = dom(1:m,n-halo+1:n) ! to North V(1:m,n+1:n+halo)[myP,SS] = dom(1:m,1:halo) ! to South V(1-halo:0,1-halo:0)[EE,NN] = dom(m-halo+1:m,n-halo+1:n) ! to North-East V(m+1:m+halo,1-halo:0)[WW,NN] = dom(1:halo,n-halo+1:n) ! to North-West V(1-halo:0,n+1:n+halo)[EE,SS] = dom(m-halo+1:m,1:halo) ! to South-East V(m+1:m+halo,n+1:n+halo)[WW,SS] = dom(1:halo,1:halo) ! to South-West sync all ! ! Now run a stencil filter over the internal region (the region unaffected by halo values) ! do j=1,n do i=1,m sum = 0. do l=-halo,halo do k=-halo,halo sum = sum + stencil(k,l)*V(i+k,j+l) enddo enddo dom(i,j) = sum enddo enddo

  8. Stencil 2D Results on XT5, XE6, X2; Halo = 1 • Using a fixed size virtual PE topology, vary the size of the local square • XT5: CAF puts/gets implemented through message-passing library • XE6, X2: RMA-enabled hardware support for PGAS, but still must pass through software stack

  9. Stencil 2D Weak Scaling on XE6 Fixed local dimension, vary the PE virtual topology (take the optimal configuration)

  10. No symmetries Any lattice with n sites — 2n states Need n bits to encode the state split this in two parts of m and n-m bits First part is a core index — 2m cores Second part is a state index within the core — 2n-m states Sparse matrix times dense vector Each process communicates (large vectors) only with m ‘neighbors’ Similar to a halo update, but with higher dimensional state space Implementation in C with MPI_Irecv/Isend, MPI_Allreduce SPIN: Transverse field Ising model Sergei Isakov 10

  11. UPC Version “Elegant” shared double *dotprod; /* on thread 0 */ shared double shared_a[THREADS]; shared double shared_b[THREADS]; structed_s { ... shared double *v0, *v1, *v2; /* vectors */ shared double *swap; /* for swapping vectors */ }; : for (iter = 0; iter < ed->max_iter; ++iter) { shared_b[MYTHREAD] = b; /* calculate beta */ upc_all_reduceD( dotprod, shared_b, UPC_ADD, THREADS, 1, NULL, UPC_IN_ALLSYNC | UPC_OUT_ALLSYNC ); ed->beta[iter] = sqrt(fabs(dotprod[0])); ib = 1.0 / ed->beta[iter]; /* normalize v1 */ upc_forall (i = 0; i < ed->nlstates; ++i; &(ed->v1[i]) ) ed->v1[i] *= ib; upc_barrier(0); /* matrix vector multiplication */ upc_forall (s = 0; s < ed->nlstates; ++s; &(ed->v1[s]) ) { /* v2 = A * v1, over all threads */ ed->v2[s] = diag(s, ed->n, ed->j) * ed->v1[s]; /* diagonal part */ for (k = 0; k < ed->n; ++k) { /* offdiagonal part */ s1 = flip_state(s, k); ed->v2[s] += ed->gamma * ed->v1[s1]; } } a = 0.0; /* Calculate local conjugate term */ upc_forall (i = 0; i < ed->nlstates; ++i; &(ed->v1[i]) ) { a += ed->v1[i] * ed->v2[i]; } shared_a[MYTHREAD] = a; upc_all_reduceD( dotprod, shared_a, UPC_ADD, THREADS, 1, NULL, UPC_IN_ALLSYNC | UPC_OUT_ALLSYNC ); ed->alpha[iter] = dotprod[0]; b = 0.0; /* v2 = v2 - v0 * beta1 - v1 * alpha1 */ upc_forall (i = 0; i < ed->nlstates; ++i; &(ed->v2[i]) ) { ed->v2[i] -= ed->v0[i] * ed->beta[iter] + ed->v1[i] * ed->alpha[iter]; b += ed->v2[i] * ed->v2[i]; } swap01(ed); swap12(ed); /* "shift" vectors */ } } POMPA Kickoff Workshop, May 3-4, 2011

  12. UPC “Inelegant1”: reproduce existing messaging • MPI MPI_Isend(ed->v1, ed->nlstates, MPI_DOUBLE, ed->to_nbs[0], k, MPI_COMM_WORLD, &req_send2); MPI_Irecv(ed->vv1, ed->nlstates, MPI_DOUBLE, ed->from_nbs[0], ed->nm-1, MPI_COMM_WORLD, &req_recv); : MPI_Isend(ed->v1, ed->nlstates, MPI_DOUBLE, ed->to_nbs[neighb], k, MPI_COMM_WORLD, &req_send2); MPI_Irecv(ed->vv2, ed->nlstates, MPI_DOUBLE, ed->from_nbs[neighb], k, MPI_COMM_WORLD, &req_recv2); : • UPC shared[NBLOCK] double vtmp[THREADS*NBLOCK]; : for (i = 0; i < NBLOCK; ++i) vtmp[i+MYTHREAD*NBLOCK] = ed->v1[i]; upc_barrier(1); for (i = 0; i < NBLOCK; ++i) ed->vv1[i] = vtmp[i+(ed->from_nbs[0]*NBLOCK)]; : for (i = 0; i < NBLOCK; ++i) ed->vv2[i] = vtmp[i+(ed->from_nbs[neighb]*NBLOCK)]; upc_barrier(2); : sPOMPA Kickoff Workshop, May 3-4, 2011

  13. UPC “Inelegant3”: use only PUT operations shared[NBLOCK] double vtmp1[THREADS*NBLOCK]; shared[NBLOCK] double vtmp2[THREADS*NBLOCK]; : upc_memput( &vtmp1[ed->to_nbs[0]*NBLOCK], ed->v1, NBLOCK*sizeof(double) ); upc_barrier(1); : if ( mode == 0 ) { upc_memput( &vtmp2[ed->to_nbs[neighb]*NBLOCK], ed->v1, NBLOCK*sizeof(double) ); } else { upc_memput( &vtmp1[ed->to_nbs[neighb]*NBLOCK], ed->v1, NBLOCK*sizeof(double) ); } : if ( mode == 0 ) { for (i = 0; i < ed->nlstates; ++i) { ed->v2[i] += ed->gamma * vtmp1[i+MYTHREAD*NBLOCK]; } mode = 1; } else { for (i = 0; i < ed->nlstates; ++i) { ed->v2[i] += ed->gamma * vtmp2[i+MYTHREAD*NBLOCK]; } mode = 0; } upc_barrier(2); POMPA Kickoff Workshop, May 3-4, 2011

  14. But then: why not use light weight SHMEM protocol? #include <shmem.h> : double *vtmp1,*vtmp2; : vtmp1 = (double *) shmalloc(ed->nlstates*sizeof(double)); vtmp2 = (double *) shmalloc(ed->nlstates*sizeof(double)); : shmem_double_put(vtmp1,ed->v1,ed->nlstates,ed->from_nbs[0]); /* Do local work */ shmem_barrier_all(); : shmem_double_put(vtmp2,ed->v1,ed->nlstates,ed->from_nbs[0]); : for (i = 0; i < ed->nlstates; ++i) { ed->v2[i] += ed->gamma * vtmp1[i]; } shmem_barrier_all(); swap(&vtmp1, &vtmp2); : Thursday, February 3, 2011 SCR discussion of HP2C projects

  15. Strong scaling: Cray XE6/Gemini, n=22,24; 10 iter. POMPA Kickoff Workshop. May 3-4, 2011

  16. Weak scaling: Cray XE6/Gemini,10 iterations POMPA Kickoff Workshop, May 3-4, 2011

  17. Conclusions • One-way communication has conceptual and can have real benefits (e.g., Cray T3E, X1, perhaps X2) • On XE6, CAF/UPC formulation can achieve SHMEM performance, but only by using puts and gets, but ‘elegant’ implementations have poor performance • If the domain decomposition is already properly formulated… why not use a simple, light-weight protocol like SHMEM?? • For XE6 Gemini interconnect: study of one-sided communication primitives (Tineo, et al.) indicates 2-sided MPI communication is still most effective. To do: test MPI-2 one-sided primitives • Still: PGAS path should be kept open; possible task: PGAS (CAF or SHMEM) implementation of COSMO halo update? POMPA Kickoff Workshop, May 3-4, 2011

More Related