1 / 16

Matrix sparsification and the sparse null space problem

Matrix sparsification and the sparse null space problem. Lee-Ad Gottlieb Weizmann Institute Tyler Neylon Bynomial Inc. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A. Matrix sparsification.

hachi
Download Presentation

Matrix sparsification and the sparse null space problem

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Matrix sparsification and the sparse null space problem Lee-Ad Gottlieb Weizmann Institute Tyler Neylon Bynomial Inc. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

  2. Matrix sparsification • Problem definition: Given a matrix, make it as sparse as possible (minimize number of non-zeros), using elementary row reductions • we want lots of 0’s • Could try Gaussian elimination… but can we do better? • Applications: • Computational speed-up for many fundamental matrix operations • Machine learning [SS-00] • Discovery of cycle bases of graphs [KMMP-04] Matrix sparsification and the sparse null space problem

  3. Matrix sparsification • What’s known about matrix sparsification? • Precious little… mostly work by McCormick and coauthors • It’s NP-hard [M-83] • Known results • Heuristic [CM-02] • Algorithm under limiting condition: [HM-84] gave an approximation algorithm for matrices that satisfy the Haar condition Matrix sparsification and the sparse null space problem

  4. Sparse null space problem • First recall the definition of null space: • The null space of A is the set of all nonzero vectors b for which Ab=0 • A null matrix for A spans the null space of A. Finding such a matrix is a basic function in linear algebra. • Problem definition: Given a matrix A, find an optimally sparse matrix N that is a full null matrix for A • N is full rank • Columns of N span the null space of A • N is sparse • Applications • Helps solve Linear Equality Problems [CP-86] (optimization via gradient descent, dual variable method for Navier-Stokes, quadratic programming) • Efficient solution to the force method for structural analysis [GH-87] • Finds correlations between time series, such as financial stocks [ZZNS-05] Matrix sparsification and the sparse null space problem

  5. Sparse null space problem • What’s known about sparse null space? • Precious little… • First considered in [P-84], it’s known to be NP-hard [CP-86] • Known results: Heuristics [BHKLPW-85, CP-86, GH-87] Matrix sparsification and the sparse null space problem

  6. Two matrix problems • It’s not difficult to see that matrix sparsification and sparse null space are equivalent • Let B be a full null matrix for A. The following statements are equivalent: • N = BX for some invertible matrix X • N is a full null matrix for X • Surprisingly then, these two lines of work have proceeded independently Matrix sparsification and the sparse null space problem

  7. Our contribution • Two matrix problems • Have been around since the 80’s • Have many applications • Are equivalent – from now on, we’ll refer only to matrix sparsification • Can we prove something concrete about matrix sparsification? • Hardness of approximation? • Approximation algorithms? • We can do both… • Hardness of approximation • As hard as label cover (quite hard to approximate) • Hard to approximate within factor 2log.5-o(1)n of optimal (with some caveats…) • Approximation algorithms • For the hard problem • Under limiting assumptions Matrix sparsification and the sparse null space problem

  8. Min unsatisfy • As a first step towards proving hardness of approximation, we’ll show that matrix sparsification is closely related to the min unsatisfy problem introduced in [ABSZ-97] • Problem definition: Given a linear system Ax=b, provide a vector x that minimizes the number of equations not satisfied • What’s known about min unsatisfy • As hard to approximate as label cover [ABSZ-97] • Under Q, hard within factor 2log.5-o(1)n of optimal under the assumption that NP does not admit a quasi-polynomial time solution. • Randomized Θ(m/log m) approximation algorithm (m is number of rows) [BK-01] x = Matrix sparsification and the sparse null space problem

  9. Hardness of matrix sparsification • We’ll give a reduction from min unsatisfy to matrix sparsification, which will prove hardness of approximation for matrix sparsification. • Preliminary note: There exist matrices that are unsparsifiable. and these can be construction in poly time. • M = (I X), where I is the identity matrix and X contains no 0 entries. • The identity portion can always be achieved via Gaussian elimination Matrix sparsification and the sparse null space problem

  10. Hardness of matrix sparsification • Proof outline • Let (A,y) be an instance of min unsatisfy • We’ll create a matrix M with a few copies of A, but many of copies of y • Minimizing the number of non-zero entries in M reduces to finding a sparse linear combination of y with some vectors of A • That is, solving the instance of min unsatisfy. • Construction: Let (Iq X) be an unsparsifiable matrix, and Ø be the Kronecker product • We chose q=n2 Matrix sparsification and the sparse null space problem

  11. Approximation algorithm • Our first result: We conclude that matrix sparsification is as hard as min unsatisfy, which itself is as hard as label cover. • Matrix sparsification is hard to approximate within factor 2log.5-o(1)n of optimal • So what can be done for matrix sparsification? • We will further show that a solution to min unsatisfy implies a similar solution for matrix sparsification. • Hence, the randomized Θ(m/log m) approximation algorithm for min unsatisfy [BK-01] carries over to matrix sparsification. • More importantly, we will also show how to extend a large number of heuristics and algorithms under limiting assumptions to apply to min unsatisfy, and therefore to matrix sparsification. • In particular, we’ll show that the well-known l1 minimization heuristic applies to matrix sparsification. Matrix sparsification and the sparse null space problem

  12. Another look at min unsatisfy • Consider the exact dictionary representation (EDR) problem, the major problem in sparse approximation theory. • Problem definition: Given a matrix D of dictionary vectors and a target vector s • Find the smallest subset D’ such that a linear combinations of vectors is equal to s. • What’s known about this problem • A variant appeared in a paper of Schmidt in 1907 [T-03] • NP-Hard [N-95] • Applications in signal representation [CW-92,NP-09], amplitude optimization [S-90], function approximation [N-95], and data mining [CRT-06, ZGSD-06, GGIMS-02, GMS-05]. • A large number of heuristics have been studied for this problem • Also approximation algorithms under limiting assumptions Matrix sparsification and the sparse null space problem

  13. Another look at min unsatisfy • EDR is in fact equivalent to min-unsatisfy ([AK-95] proved one direction) although this seems to have escaped the notice of the sparse approximation theory community. • We’ll show how to extend the heuristics and algorithms for EDR (and therefore, min unsatify) for matrix sparsification. Matrix sparsification and the sparse null space problem

  14. Matrix sparsification • The following greedy algorithm solves matrix sparsification • We assumes existence of subroutine SIV(A,B), • returns the sparsest vector in the span of matrix A that is not in the span of matrix B • Notes: • This subroutine can be easily implemented using a heuristic or approximation algorithm for min unsatisfy (see paper) • The matrix sparsification algorithm below is a slight simplification (again see paper) • Algorithm for matrix sparsification builds matrix B one column at a time • B ← null • For i=n…1 • a = SIV(A,B) • B ← a • [CP-86] proved that the greedy algorithm works for matroids. Matrix sparsification and the sparse null space problem

  15. Algorithms for matrix sparsification • We conclude that all algorithms for min unsatisfy (and EDR) apply to matrix sparsification as well. • There exists a randomized Θ(m/log m) approximation algorithm for matrix sparsification. • A large number of heuristics for EDR carry over to matrix sparsification. • Practical contribution • The popular l1 minimization heuristic for EDR carries over to matrix sparsification • This heuristic finds a vector v that satisfies Dv=s, while minimizing ||v||1 instead of number of non-zeros in v • The heuristic is also an approximation algorithm under certain limiting assumptions [F-04] • This heuristic for matrix sparsification has already been implemented since the public posting of our paper! Matrix sparsification and the sparse null space problem

  16. Conclusion • Considered the matrix sparsification and sparse null space problems. • Showed that they are very hard to approximate. • Showed how to extend a large number of studied heuristics and algorithms to these problems Matrix sparsification and the sparse null space problem

More Related