Solution of Sparse Linear Systems

1. Numerical Simulation 219A Spring 2008 Solution of Sparse Linear Systems Thanks to: Kin Sou, Deepak Ramaswamy, Michal Rewienski, Jacob White, Shihhsien Kuo and Karen Veroy and especially Luca Daniel

2. Outline of today’s lecture • Solution of Sparse Linear Systems • Examples of Problems with Sparse Matrices • Struts and joints, resistor grids, 3-D heat flow • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Sparse Matrix Data Structures • Scattering

3. 5 7 9 3 4 6 8 2 1 Applications Sparse Matrices Space Frame Nodal Matrix Space Frame X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Unknowns : Joint positions Equations : forces = 0

4. Applications Sparse Matrices Resistor Grid Unknowns : Node Voltages Equations : currents = 0

5. Applications Sparse Matrices Nodal Formulation Resistor Grid Matrix non-zero locations for 100 x 10 Resistor Grid

6. Applications Sparse Matrices Nodal Formulation Temperature in a cube Temperature known on surface, determine interior temperature Circuit Model

7. Outline • Solution of Dense Linear Systems • Solution of Sparse Linear Systems • LU Factorization Reminder. • Example of Problems with Sparse Matrices • Struts and joints, resistor grids, 3-d heat flow • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Sparse Matrix Data Structures • Scattering

8. Tridiagonal Example Sparse Matrices Nodal Formulation Matrix Form How many operations to do LU factorization? m

9. Pivot Multiplier Tridiagonal Example Sparse Matrices GE Algorithm For i = 1 to n-1 {“For each Row” For j = i+1 to n {“For each target Row below the source” For k = i+1 to n {“For each Row element beyond Pivot” } } } Order n Operations!

10. Outline • Solution of Dense Linear Systems • Solution of Sparse Linear Systems • LU Factorization Reminder. • Example of Problems with Sparse Matrices • Struts and joints, resistor grids, 3-d heat flow • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Sparse Matrix Data Structures • Scattering

11. Sparse Matrix Fill-In Sparse Matrices Example Resistor Example Nodal Matrix 0 Symmetric Diagonally Dominant

12. X X X = fill in where 0 became non-zero Fill-In Sparse Matrices Example Matrix Non zero structure Fill Ins X X X X X= Non zero

13. X X X X X Fill-In Sparse Matrices Second Example Fill-ins Propagate X X X X X Fill-ins from Step 1 result in Fill-ins in step 2

14. 0 Fill-In Sparse Matrices Reordering Fill-ins 0 No Fill-ins Node Reordering - Can reduce fill-in - Preserves properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns

15. Possible Fill-in Locations Fill-in Estimate = (Non zeros in unfactored part of Row -i) (Non zeros in unfactored part of Col -i) Markowitz product Fill-In Sparse Matrices Reordering Where can fill-in occur ? Already Factored Multipliers

16. Fill-In Sparse Matrices Reordering Markowitz Reordering Greedy Algorithm !

17. 1 2 3 1 0 0 3 2 Fill-In Sparse Matrices Reordering Why only try diagonals ? • Corresponds to node reordering in Nodal formulation • Reduces search cost • Preserves Matrix Properties - Diagonal Dominance - Symmetry

18. Fill-In Sparse Matrices Pattern of a Filled-in Matrix Very Sparse Very Sparse Dense

19. Fill-In Sparse Matrices Unfactored Random Matrix Symmetric

20. Fill-In Sparse Matrices Factored Random Matrix Symmetric

21. Outline • Solution of Dense Linear Systems • Solution of Sparse Linear Systems • LU Factorization Reminder. • Example of Problems with Sparse Matrices • Struts and joints, resistor grids, 3-d heat flow • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Sparse Matrix Data Structures • Scattering

22. Matrix Graphs Sparse Matrices Construction Structurally Symmetric Matrices and Graphs 1 X X X X X X X 2 X X X X 3 X X X 4 X X X 5 • One Node Per Matrix Row • One Edge Per Off-diagonal Pair

23. 1 2 X X X X 3 X X X 4 X X X X 5 X X X X X X Matrix Graphs Sparse Matrices Markowitz Products Markowitz Products = (Node Degree)2

24. Matrix Graphs Sparse Matrices Factorization One Step of LU Factorization 1 X X X X X X X X 2 X X X X X 3 X X X 4 X X X X X 5 • Delete the node associated with pivot row • “Tie together” the graph edges

25. 1 2 3 4 5 1 2 3 4 5 Matrix Graphs Sparse Matrices Example Graph Markowitz products ( = Node degree)

26. 5 5 4 3 2 3 4 Graph Matrix Graphs Sparse Matrices Example Swap 2 with 1 X 1 2 2 1 X Eliminate 1

27. Graphs Sparse Matrices Resistor Grid Example Unknowns : Node Voltages Equations : currents = 0

28. Matrix Graphs Sparse Matrices Grid Example

29. What should you pivot for? Sparse Matrices For growth control? Or to avoid fill-ins? A) LU factorization applied to a strictly diagonally dominant matrix will never produce a zero pivot B) The matrix entries produced by LU factorization applied to a strictly diagonally dominant matrix will never increase by more than a factor 2(n-1) [which is anyway the best you can do by pivoting for growth control] Bottom line: if your matrix is strictly diagonally dominant no need for numerical pivot for growth control!

30. Sparse Factorization Approach • Assume matrix requires NO numerical pivoting. Diagonally dominant or symmetric positive definite. • Pre-Process: • Use Graphs to Determine Matrix Ordering • Many graph manipulation tricks used. • Form Data Structure for Storing Filled-in Matrix • Lots of additional non-zero added • Put numerical values in Data Structure and factor Computation must be organized carefully!

31. Summary of Sparse Systems • LU Factorization and Diagonal Dominance. • Factor without numerical pivoting • Sparse Matrices • Struts, resistor grids, 3-d heat flow -> O(N) non-zeros • Tridiagonal Matrix Factorization • Factor in O(N) operations • General Sparse Factorization • Markowitz Reordering to minimize fill • Graph Based Approach • Factorization and Fill-in • Useful for estimating Sparse GE complexity

32. Outline • Solution of Sparse Linear Systems • Example of Problems with Sparse Matrices • Tridiagonal Matrix Factorization • General Sparse Factorization • Fill-in and Reordering • Graph Based Approach • Summary of the Algorithm • Sparse Matrix Data Structures • Scattering and symbolic fill in

33. Sparse Data Structure Sparse Matrices Vector of row pointers Arrays of Data in a Row Matrix entries Val 11 Val 12 Val 1K 1 Column index Col 11 Col 12 Col 1K Val 21 Val 22 Val 2L Col 21 Col 22 Col 2L Row pointers Column Indices Goal: Never store a 0 Never multiply by 0 Val N1 Val N2 Val Nj N Col N1 Col N2 Col Nj

34. Sparse Data Structure Sparse Matrices Problem of Misses Eliminating Source Row i from Target row j Row i Row j Must read all the row j entries to find the 3 that match row i

35. Sparse Data Structure Sparse Matrices Data on Misses More misses than ops! Every Miss is an unneeded Memory Reference!

36. Sparse Data Structure Sparse Matrices Scattering for Miss Avoidance Row j Use target row approach – Row j is the target. 1) Read all the elements in Row j, and scatter them in an n-length vector 2) Access only the needed elements using array indexing!

37. Sparse Matrices – Another Data Structure Orthogonal linked list But if fill in occurs, pointer structures change. What do we do? Pre-compute pivoting order and sparse fill in structure symbolically

38. Summary of Sparse Systems • LU Factorization and Diagonal Dominance. • Factor without numerical pivoting • hard problems (ill-conditioned) • Sparse Matrices • Struts, resistor grids, 3-d heat flow -> O(N) nonzeros • Tridiagonal Matrix Factorization • Factor in O(N) operations • General Sparse Factorization • Markowitz Reordering to minimize fill • Graph Based Approach • Factorization and Fill-in • Useful for estimating Sparse GE complexity • Sparse Data Structures • Scattering and symbolic fill in