230 likes | 350 Views
Dimension Reduction using Rademacher Series on Dual BCH Codes. Nir Ailon Edo Liberty. Dimension Reduction. Algorithmic technique: often as “black box” Removes redundancy from data A sampling method Measure concentration phenomenon (Existential) metric embedding
E N D
Dimension ReductionusingRademacher SeriesonDual BCH Codes Nir Ailon Edo Liberty
Dimension Reduction • Algorithmic technique: • often as “black box” • Removes redundancy from data • A sampling method • Measure concentration phenomenon • (Existential) metric embedding • Recently: computational aspects • time • randomness
x x Rd Rk Challenge Find random projection from Rd to Rk (d big, k small) such that for every x Rd,||x||2=1, 0< with probability exp{-k ||x||2 = 1 ± O(
x1 x1 x3 x2 x2 x3 Rd Rk Usage If you have n vectors x1..xn Rd: set k=O(log n) by union bound: for all i,j ||xi- xj|| ≈||xi- xj|| low-distortion metric embedding “tight”
Solution: Johnson-Lindenstrauss (JL) “dense random matrix” = k d
So what’s the problem? • running time (kd) • number of random bits (kd) • can we do better?
Fast JL A, Chazelle 2006 • = Sparse.Hadamard.Diagonal time = O(k3 + dlog d), randomness=O(k3 + d) beats JL (kd) bound for: log d < k < d1/3 k Fourier d
This Talk • O(d logk) for k < d1/2 • beats JL up to k < d1/2 • O(d) random bits
Algorithm (k=d1/2) A, Liberty 2007 • = BCH .Diagonal . .... Error Correcting Code k d
binary “dual-BCH code of designed distance 4” Bx computable in time d logd given x Error Correcting Codes d B = k = d1/2 0 k-1/2 1 -k-1/2 columns closed under addition row set subset of d x d Hadamard 4-wise independence
Error Correcting Codes Fact (easily from properties of dual BCH): ||Bt||24 = O(1) for yt Rk with ||y||2 = 1: ||yB||4 = O(1) x x ... Rd D B Rk=d
Rademacher Serieson Error Correcting Codes look at r.v. BDx (Rk, l2) BDx = DiixiBi Dii R{1} i=1...d = DiiMi x x ... Rd D B Rk=d
Talagrand’s Concentration Boundfor Rademacher Series Z =||DiiMi||p (in our case p=2) Pr[ |Z-EZ| > ] = O(exp{-2/4||M||2p2})
Rademacher Serieson Error Correcting Codes look at r.v. BDx (Rk, l2) Z = ||BDx||2 = ||DiiMi||2 ||M||22||x||4||Bt||24 (Cauchy-Schwartz) by ECC properties: ||M||22||x||4O(1) trivial: EZ = ||x||2 = 1 x x ... Rd D B Rk=d
Rademacher Serieson Error Correcting Codes look at r.v. BDx (Rk, l2) Z = ||BDx||2 = ||DiiMi||2 ||M||22=O(||x||4) EZ = 1 Pr[ |Z-EZ| > ] = O(exp{-2/4||M||222}) • Pr[|Z-1| > ] = O(exp{-2/||x||42}) how to get ||x||42 = O(k-1=d-1/2) ? x x Challenge: w. prob. exp{-k deviation of more than ... Rd D B Rk=d
Controlling ||x||42 how to get ||x||42 = O(k-1=d-1/2) ? • if you think about it for a second... • “random” x has ||x||42=O(d-1/2) • but “random” x easy to reduce: just output first k dimensions • are we asking for too much? • no: truly random x has strong bound on||x||p for all p>2 x x ... Rd D B Rk=d
Controlling ||x||42 how to get ||x||42 = O(k-1=d-1/2) ? • can multiply x by orthogonal matrix • try matrix HD • Z = ||HDx||4= ||DiixiHi||4 =||DiiMi||4 • by Talagrand:Pr[ |Z-EZ| > t ] = O(exp{-t2/4||M||242}) EZ = O(d-1/4) (trivial) ||M||24||H||4/34||x||4(Cauchy Schwartz) (HD used in [AC06] to control ||HDx||) x x ... Rd D B Rk=d
x x ... Rd D B Rk=d Controlling ||x||42 how to get ||x||42 = O(k-1=d-1/2) ? Z =||DiiMi||4 Mi = xiHi Pr[ |Z-EZ| > t ] = O(exp{-t2/4||M||242}) (Talagrand)EZ = O(d-1/4) (trivial)||M||24||H||4/34||x||4 (Cauchy Schwartz) ||H||4/34 d-1/4 (Hausdorff-Young) • ||M||24d-1/4||x||4 • Pr[ ||HDx||4 > d-1/4+ t ] = exp{-t2/d-1/2||x||42}
x x ... Rd D B Rk=d Controlling ||x||42 how to get ||x||42 = O(d-1/2) ? Pr[ ||HDx||4 > d-1/4+ t ] = exp{-t2/d-1/2||x||42} need some slack k=d-1/2- max error probability for challenge: exp{-k} k = t2/d-1/2||x||42 t = k1/2d1/4||x||4 = ||x||4d-/2
x x ... Rd D B Rk=d Controlling ||x||42 how to get ||x||42 = O(d-1/2) ? first round: ||HDx||4 < d-1/4 + ||x||4 d-/2 second round: ||HD’’HD’x||4 < d-1/4 + d-1/4-/2 + ||x||4 d- r=O(1/)’th round: ||HD(r)...HD’x||4 < O(r)d-1/4 ... with probability 1-O(exp{-k})
Algorithm for k=d-1/2- x x ... Rk Rd D(1) H D(r) H D B running time O(d logd) randomness O(d)
Open Problems • Go beyond k=d1/2Conjecture: can do O(d log d) for k=d1- • Approximate linear l2-regressionminimize ||Ax-b||2 given A,b (overdetermined) • State of the art for general inputs: Õ(linear time) for #{variables} < #{equations}1/3 • Conjecture: can do Õ(linear time) always