CADD : Component-Averaged Domain Decomposition

CADD:Component-AveragedDomain Decomposition Dan Gordon Computer Science University of Haifa Rachel Gordon Aerospace Engg. Technion Component-Averaged Domain Decomposition

Outline of Talk • CADD – Algebraic explanation • CADD – DD explanation • The Kaczmarz algorithm (KACZ) • KACZ CARP (a CADD method) • Applications of CARP • CARP-CG: CG acceleration of CARP • Sample results Component-Averaged Domain Decomposition

CADD – Algebraic Explanation – 1 • The equations are divided into blocks (not necessarily disjoint) • Initial estimate: vector x=(x1,…,xn) • Suppose x1 is a variable (component of x) that appears in 3 blocks • x1 is “cloned” as y1 , z1 , t1 in the different blocks. • Perform one (or more) iterative operation(s) on each block (independently, in parallel) Component-Averaged Domain Decomposition

CADD – Algebraic Explanation – 2 • The internal iterations in each block produce 3 new values for the clones of x1: y1’ , z1’ , t1’ • The next iterative value of x1 is x1’ = (y1’ + z1’ + t1’)/3 • The next iterate is x’ = (x1’ , ... , xn’) • Repeat iterations as needed for convergence Component-Averaged Domain Decomposition

CADD – non-overlapping domains clone of x1 y 1 x x 1 0 external grid point of A domain A domain B Component-Averaged Domain Decomposition

CADD – overlapping domains domain B clone of x1 y x 1 x 0 1 external grid point of A A∩B domain A Component-Averaged Domain Decomposition

CADD – Parallel Implementation • Every processor is in charge of one domain • Mode: BLOCK-PARALLEL • All processors operate in parallel (each on its domain) • Processors exchange clone values • All values are updated • New values = average of clones Component-Averaged Domain Decomposition

CADD Vs. Standard DD Methods • Difference is in handling external grid points: • CADD: A clone of the external grid point is modified by the domain's equations and contributes to new value of grid point. • Standard DD: Value of external grid point is fixed (contributes to RHS) Component-Averaged Domain Decomposition

KACZ: The Kaczmarz algorithm • Iterative method, due to Kaczmarz (1937). Rediscovered for CT as ART • Basic idea: Consider the hyperplane defined by each equation • Start from an arbitrary initial point • Successively project current point onto the next hyperplane, in cyclic order Component-Averaged Domain Decomposition

KACZ: Geometric Description initial point eq. 1 eq. 2 eq. 3 Component-Averaged Domain Decomposition

KACZ with Relaxation Parameter • KACZ can be used with a relaxation parameter λ • λ=1: project exactly on the hyperplane • λ<1: project in front of hyperplane • λ>1: project beyond the hyperplane • Cyclicrelaxation: Eq. i is assigned a relaxation parameter λi Component-Averaged Domain Decomposition

Convergence Properties of KACZ • KACZ with relaxation (0< λ <2)converges for consistent systems: • Herman, Lent & Lutz, 1978 • Trummer, 1981 • For inconsistent systems, KACZ converges cyclically: • Tanabe, 1971 • Eggermont, Herman & Lent, 1981 (for cyclic relaxation parameters). Component-Averaged Domain Decomposition

Another view of KACZ • Given the system Ax = b • Consider the "normal equations" system AATy = b, x=ATy • Well-known: KACZ is simply SOR applied to the normal equations • The relaxation parameter of KACZ is the usual relax. par. of SOR Component-Averaged Domain Decomposition

CARP: Component-Averaged Row Projections • CARP is a CADD method with KACZ iterations in each domain: • The system Ax=b is divided into blocks B1,…,Bn (need not be disjoint) • Each processor is assigned one (or more) block • All the blocks are processed in parallel • Results from blocks are “merged” to form the next iterate • Merging is done by CADD averaging Component-Averaged Domain Decomposition

Overview of CARP domain A domain B averaging KACZ iterations KACZ iterations cloning KACZ in superspace (with cyclic relaxation) Component-Averaged Domain Decomposition

Convergence of CARP • Averaging Lemma: the component-averaging and cloning operations of CARP are equivalent to KACZ row-projections in a certain superspace (with λ= 1) • CARP is equivalent to KACZ in the superspace, with cyclic relaxation parameters – known to converge Component-Averaged Domain Decomposition

"Historical" Note • The term "component averaging" was first used by Censor, Gordon & Gordon (2001) for CAV and BICAV, used for image reconstruction in CT • These are Cimmino-type algorithms, with weights related to the sparsity of the system matrix • CADD, CARP use this concept in a different sense Component-Averaged Domain Decomposition

CARP Application: Solution of stiff linear systems from PDEs • Elliptic PDEs w/large convection term result in stiff linear systems (large off-diagonal elements) • CARP is very robust on these systems, as compared to leading solver/preconditioner combinations • Downside: Not always efficient Component-Averaged Domain Decomposition

CARP Application: Electron Tomography(joint work with J.-J. Fernández) • 3D reconstructions: Each processor is assigned a block of consecutive slices. Data is in overlapping blobs. • The blocks are processed in parallel. • The values of shared variables are transmitted between the processors which share them, averaged, and redestributed. Component-Averaged Domain Decomposition

CARP-CG: CG acceleration of CARP • CARP is KACZ in some superspace (with cyclic relaxation parameter) • Björck & Elfving (BIT 79): developed CGMN, which is a (sequential)CG-acceleration of KACZ (double sweep, fixed relax. parameter) • We extended this result to allow cyclic relaxation parameters • Result: CARP-CG Component-Averaged Domain Decomposition

CARP-CG: Properties • Same robustness as CARP • Very significant improvement in performance on stiff linear systems derived from elliptic PDEs • Very competitive runtime compared to leading solver/preconditioner combinations on systems derived from convection-dominated PDEs • Improved performance in ET Component-Averaged Domain Decomposition

CARP-CG on PDEs CG acceleration of projection methods was done before, but with block projections, requiring multi-coloring. CARP-CG avoids this. Tests were run on 9 convection-dominated PDEs, comparing CARP-CG with restarted GMRES, CGS and Bi-CGSTAB, with and without various preconditioners. Also tested: CGNR. Domain: unit cube – 80x80x80. Component-Averaged Domain Decomposition

512,000 equations Component-Averaged Domain Decomposition

CARP-CG: ET results Component-Averaged Domain Decomposition

Future research on CADD • Different internal solvers: 1. Guaranteed convergence? 2. Symmetrizability? • Combine CADD with multigrid • CFD applications • Additional biomedical applications • CADD with preconditioners • Compare CADD with other DD methods, such as additive Schwarz Component-Averaged Domain Decomposition

CARP Publications • CARP: SIAM J. Sci. Comput. 27 (2005) 1092-1117. http://cs.haifa.ac.il/~gordon/carp.pdf • Electron tomography: J. Par. & Dist. Comput. (2007), in press. http://cs.haifa.ac.il/~gordon/eltom.pdf • CARP-CG: Submitted for publication. Component-Averaged Domain Decomposition

CADD : Component-Averaged Domain Decomposition