1 / 10

=SpS ( XX,

=SpS ( XX,. p 23 +p 23 p 22 +. p 33 +p 33 p 32 +. p 43 +p 43 p 42 -2. p 13 p 33 -2p 13 p 32 -2. p 23 p 43 -2p 23 p 42 ) +. p 13 +p 13 p 12 +. 2 6 (. p 23 p 21 +. p 33 p 31 +. p 43 p 41 -2. p 13 p 31 -2. p 23 p 41 ) +. p 13 p 11 +. 2 5 (. p 12 +p 13 p 10 +p 12 p 11 +.

dsilva
Download Presentation

=SpS ( XX,

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. =SpS(XX, p23+p23p22 + p33+p33p32 + p43+p43p42 -2 p13p33-2p13p32 -2 p23p43-2p23p42 ) + p13+p13p12 + 26( p23p21 + p33p31 + p43p41 -2 p13p31 -2 p23p41 ) + p13p11 + 25( p12+p13p10+p12p11 + p22+p23p20+p22p21 + p42+p43p40+p42p41 -2p22p42-2p23p40-2p22p41) + p32+p33p30+p32p31 + -2p12p32-2p13p30-2p12p31 24( p22p20 + p32p30 + p42p40 -2 p12p30 -2 p22p40 ) + p12p10 + 23( -2p11p31-2p11p30 -2 p21p41-2p21p40 ) + p10 + p20 + p30 + p40 -2p10p30 -2p20p40) p11+p11p10 + p21+p21p20 + p31+p31p30 + p41+p41p40 22( p13 p12 p11 p10 p23 p22 p21 p20 p33 p32 p31 p30 p43 p42 p41 p40 * 1 1 1 1 1 1 1 * 1 1 1 1 1 * 1 1 1 * 1 * 1 1 1 1 1 1 1 * 1 1 1 1 1 * 1 1 1 * 1 * 1 1 1 * 1 1 * 1 * * 1 1 1 * 1 1 * 1 * p13 p12 p11 p10 p23 p22 p21 p20 p33 p32 p31 p30 p43 p42 p41 p40 p13p33 + p13p32 + p23p43 p23p42 -27( =SpS(XX, + )+ p13p31 + p23p41 p13+p23+p33+p43 )+ +p13p12+p23p22 + +p33p32 + +p43p42 ) -26( 26( +p23p40 p23p21 + p33p31 + p43p41 ) p13p11+ -25( p13p30 25( +p12p31 +p22p41 +p12p32 +p22p42 )+ p12+p22+p32+p42 +p23p20 +p43p40 -24(p12p30 )+ +p13p10+ +p33p30 +p22p40 24( +p32p31 +p42p41 ) +p12p11+ +p22p21 p22p20 + p32p30 + p42p40 ) p12p10+ -23(p11p31 +p11p30 +p21p41 +p21p40 23( )+ p11+p21+p31+p41 +p11p10 + +p21p20 + +p31p30 -22(p10p30+p20p40 +p41p40 ) )+ 22( p10+p20+p30+p40) The computation for SpS(XX,d2(x,y)) shown below is chock full of useful information, but is also massive and time consuming to compute (even with 2 level pTrees). Can we find a simpler functional involving some subset of the 40 terms below that is square distance dominated? SpS[ XX, d2(x=(x1,x2), y=(x3,x4)] = SpS[ XX, (x1-x3)2+(x2-x4)2 ] = SpS(XX, x1x1 + x2x2 + x3x3 + x4x4 - 2x1x3 -2x2x4) =SpS(XX, p13p33+p13p32 +p23p43+p23p42 )) + - p13p31 - p23p41 p13+p23+p33+p43 +p13p12 + -2( +p23p22 + +p33p32 + +p43p42 26( p12+p13p10+p12p11 + p22+p23p20+p22p21 + p42+p43p40+p42p41 - p12p30 - p22p40 p32+p33p30+p32p31 + 24( p23p21 + p33p31 + p43p41 p13p11 + - p12p32 - p13p30 - p12p31 - - p22p42 - p23p40 - p22p41 )) + +2( p11+p11p10 + p21+p21p20 + p31+p31p30 + - p10p30 -p20p40 p41+p41p40 ) + 22( p10 + p20 + p30 + p40 p22p20 + p32p30 + p42p40 - )) + p12p10 + p11p31- p11p30 - p21p41- p21p40 +2( piipii=pii (no processing) Only 44 the pairwise products need computing.

  2. The computation for SpS(XX,d2(x,y)) is massive and time consuming (even with 2 level pTrees). A Manhattan_distance based functional may be simpler and nearly as useful? L1(x,y)≡i=1..n|xi-yi| Claim L1(x,y)  √2L2(x,y) ≡ √2(i=1..n(xi-yi)2)½ Proof: Let x2+y2=1, f(x,y) = x+y , so f(x)= x + (1-x2)½ f'(x) = 1 + ½(1-x2)-½ (-2x) = 0 1 = x / (1-x2)½ 1 = x2 / 1-x2 1-x2 = x2 1= 2x2 1/2= x2 1/√2= x 1 x y The following utilities or procedures can be computed without vertical loops (pTree &s, |s, etc. only): pTree Operations: &, |, XOR, complement... input_operands=pTrees output=pTree Vertical Functionals: max, min, rankK,dot,+,-,*,... input_operand=PTreeSet (VectorSpace) output=SpS Functional Contours: MaxPts, MinPts, RanKPts, ... input_operand=vector space, functional, set of reals. output=set of points that map into that set of reals under that functional on that table.

  3. F1=Square Dis Functional is a hard pTree computation (Md?). p13+p23+p33+p43 +p13p12+p23p22+ +p33p32 + +p43p42 p13p33+p13p32+ -p13p31- p23p41) p23p43+p23p42) F1=pS( XX,(x1-x3)2+(x2-x4)2= -p32p31 -p42p41 ) -27( +26( F2=simpler distance dominated functional Manhattan-distance-based (but the masks may be as difficult to construct as the terms of F1?) p12+p22+p32+p42 +p23p20 +p43p40 -p12p30 -p12p11 -p22p21 +p13p10+ +p33p30 -p22p40 -p12p31 -p22p41 -p12p32 -p22p42)+ -p13p30 -p23p40 p23p21+ p33p31+ p13p11+ p43p41 24( +25( p11+p21+p31+p41 +p11p10 + +p21p20 + +p31p30 -p10p30-p20p40) + p10+p20+p30+p40 -p11p31-p11p30 -p21p41-p21p40 F2 +p41p40 p22p20 + p32p30 + p42p40 p12p10+ +22( )+ +23( Consider the functionals G1=(x1-x3)+(x2-x4)  L1  √2*L2. G2=(x1-x3)+(x4-x2) since L1((x1,x2),(x3,x4))= |x1-x3| + |x2-x4|  (x1-x3)+(x2-x4) etc. G3=(x3-x1)+(x2-x4) G4=(x3-x1)+(x4-x2) | Gi | / √2 is a distance dominated functional on XX, e.g., on z1: F2=SpS(XX,(Rnd((1/√2)((x1>x3&x2>x4)*(x1-x3+x2-x4)+ (x1x3&x2x4)*(x3-x1+x4-x2)))) (x1>x3&x2x4)*(x1-x3+x4-x2)+ (x1x3&x2>x4)*(x3-x1+x2-x4)+ p33 p32 p31 p30 p43 p41 p40 X3 X4 p42 X1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 : 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 X2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 p13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p20 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IDX z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 : ze ze ze ze ze ze ze ze ze ze ze ze ze ze ze zf zf zf zf zf zf zf zf zf zf zf zf zf zf zf IDY z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf p10 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 F1 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 0 4 2 8 17 68 196 170 200 153 145 181 164 200 85 4 0 2 4 5 40 144 122 148 109 113 145 136 164 65 : 200 164 162 128 117 68 116 90 80 53 5 1 4 0 25 85 65 61 41 40 29 113 85 89 52 10 20 13 25 0 0 1 1 3 4 7 10 10 11 11 12 13 13 14 9 1 0 1 1 2 6 8 8 10 9 11 12 11 13 8 : 14 13 13 11 11 7 10 8 8 6 2 1 1 0 5 9 8 8 6 6 5 11 9 9 7 3 4 4 5 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 dXY 0 2 1 3 4 8 14 13 14 12 12 13 13 14 9 ID1 ID2 G1 G2 G3 G4 MXabs z1 z1 0 0 0 0 0 z1 z2 -1 -1 1 1 1 z1 z3 -1 0 0 1 1 z1 z4 -3 0 0 3 3 z1 z5 -4 -2 2 4 4 z1 z6 -7 -4 4 7 7 z1 z7 -10 -10 10 10 10 z1 z8 -10 -8 8 10 10 z1 z9 -11 -8 8 11 11 z1 z10 -11 -6 6 11 11 z1 z11 -12 -1 1 12 12 z1 z12 -13 -1 1 13 13 z1 z13 -13 1 -1 13 13 z1 z14 -14 0 0 14 14 z1 z15 -9 1 -1 9 9 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 : 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 : 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 : 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 : 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 : z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf : 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 : 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 : 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 : 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 : 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 : 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8

  4. v=0; p=K; P=Pur1; For i=n..0 { If (c=Ct(P&P3)p) {v=v+2i;P=P&Pi} else { p=p-c;P=P&P'i}; return v, P; ID1ID2 L1XY z13z1 18 z13z2 16 z13z3 16 z13z4 14 z13z5 13 z13z6 8 z13z7 16 z13z8 14 z13z9 14 z13z10 11 z13z11 3 z13z12 3 z13z13 0 z13z14 2 z13z15 5 z14z1 20 z14z2 18 z14z3 18 z14z4 16 z14z5 15 z14z6 10 z14z7 14 z14z8 12 z14z9 12 z14z10 9 z14z11 3 z14z12 1 z14z13 2 z14z14 0 z14z15 7 z15z1 13 z15z2 11 z15z3 11 z15z4 9 z15z5 8 z15z6 7 z15z7 15 z15z8 13 z15z9 13 z15z10 10 z15z11 4 z15z12 6 z15z13 5 z15z14 7 z15z15 0 ID1ID2 L1XY z9 z1 16 z9 z2 14 z9 z3 14 z9 z4 12 z9 z5 11 z9 z6 6 z9 z7 2 z9 z8 2 z9 z9 0 z9 z10 3 z9 z11 11 z9 z12 11 z9 z13 14 z9 z14 12 z9 z15 13 z10z1 15 z10z2 13 z10z3 13 z10z4 11 z10z5 10 z10z6 5 z10z7 5 z10z8 3 z10z9 3 z10z10 0 z10z11 8 z10z12 8 z10z13 11 z10z14 9 z10z15 10 z11z1 17 z11z2 15 z11z3 15 z11z4 13 z11z5 12 z11z6 7 z11z7 13 z11z8 11 z11z9 11 z11z10 8 z11z11 0 z11z12 2 z11z13 3 z11z14 3 z11z15 4 z12z1 19 z12z2 17 z12z3 17 z12z4 15 z12z5 14 z12z6 9 z12z7 13 z12z8 11 z12z9 11 z12z10 8 z12z11 2 z12z12 0 z12z13 3 z12z14 1 z12z15 6 ID1ID2 L1XY z1 z1 0 z1 z2 2 z1 z3 2 z1 z4 4 z1 z5 5 z1 z6 10 z1 z7 14 z1 z8 14 z1 z9 16 z1 z10 15 z1 z11 17 z1 z12 19 z1 z13 18 z1 z14 20 z1 z15 13 z2 z1 2 z2 z2 0 z2 z3 2 z2 z4 2 z2 z5 3 z2 z6 8 z2 z7 12 z2 z8 12 z2 z9 14 z2 z10 13 z2 z11 15 z2 z12 17 z2 z13 16 z2 z14 18 z2 z15 11 z3 z1 2 z3 z2 2 z3 z3 0 z3 z4 2 z3 z5 3 z3 z6 8 z3 z7 14 z3 z8 12 z3 z9 14 z3 z10 13 z3 z11 15 z3 z12 17 z3 z13 16 z3 z14 18 z3 z15 11 z4 z1 4 z4 z2 2 z4 z3 2 z4 z4 0 z4 z5 3 z4 z6 6 z4 z7 14 z4 z8 12 z4 z9 12 z4 z10 11 z4 z11 13 z4 z12 15 z4 z13 14 z4 z14 16 z4 z15 9 ID1ID2 L1XY z5 z1 5 z5 z2 3 z5 z3 3 z5 z4 3 z5 z5 0 z5 z6 5 z5 z7 11 z5 z8 9 z5 z9 11 z5 z10 10 z5 z11 12 z5 z12 14 z5 z13 13 z5 z14 15 z5 z15 8 z6 z1 10 z6 z2 8 z6 z3 8 z6 z4 6 z6 z5 5 z6 z6 0 z6 z7 8 z6 z8 6 z6 z9 6 z6 z10 5 z6 z11 7 z6 z12 9 z6 z13 8 z6 z14 10 z6 z15 7 z7 z1 14 z7 z2 12 z7 z3 14 z7 z4 14 z7 z5 11 z7 z6 8 z7 z7 0 z7 z8 2 z7 z9 2 z7 z10 5 z7 z11 13 z7 z12 13 z7 z13 16 z7 z14 14 z7 z15 15 z8 z1 14 z8 z2 12 z8 z3 12 z8 z4 12 z8 z5 9 z8 z6 6 z8 z7 2 z8 z8 0 z8 z9 2 z8 z10 3 z8 z11 11 z8 z12 11 z8 z13 14 z8 z14 12 z8 z15 13 Modify the count procedure to output the count for each stride as it goes and to (fork off) compose P as it goes (by anding with the level0 stride portion of Pi or P'i), ending up with the P for the next round and the v and p for the next round (again as a forked off process as it goes. Do this with 2-level pTrees!

  5. APPENDIX pTree Rank(K)(Rank(n-1) applied to SpS(XX,d2(x,y))gives 2nd smallest distance from each x (useful in outlier analysis?) RankKval= 0 1 0 0 0 0 0 23 * + 22 * + 21 * + 20 * = 5P=MapRankKPts= ListRankKPts={2} RankKval=0; p=K; c=0; P=Pure1; /*Also RankPts are returned as the resulting pTree, P*/ For i=n to 0 {c=Count(P&Pi); If (c>=p) {RankVal=RankVal+2i; P=P&Pi}; else {p=p-c;P=P&P'i }; return RankKval, P; /* Below K=n-1=7-1=6 (looking for the 6th highest = 2nd lowest value) */ /* Notice that each new P has value. We should retain every one of them. How to catalog in 2pDoop?? */ Cross out the 0-positions of P each step. (n=3) c=Count(P&P4,3)= 3 < 6 p=6–3=3; P=P&P’4,3 masks off highest 3 (val 8) {0} X P4,3P4,2P4,1 P4,0 0 1 1 1 0 0 0 0 1 0 1 1 1 1 10 5 6 7 11 9 3 1 0 0 0 1 1 0 1 0 1 1 1 0 1 (n=2) c=Count(P&P4,2)= 3 >= 3 P=P&P4,2 masks off lowest 1 (val 4) {1} (n=1) c=Count(P&P4,1)=2 < 3 p=3-2=1; P=P&P'4,1 masks off highest 2 (val8-2=6 ) {0} {1} (n=0) c=Count(P&P4,0 )=1 >= 1 P=P&P4,0 {0} {1} {0} {1}

  6. Val=0;p=K;c=0;P=Pure1; For i=n to 0 {c=Ct(P&Pi); If (c>=p){Val=Val+2i;P=P&Pi}; else{p=p-c;P=P&P'i }; return Val, P; X3 X4 IDY z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 : 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 : z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf : 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 IDX z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 : ze ze ze ze ze ze ze ze ze ze ze ze ze ze ze zf zf zf zf zf zf zf zf zf zf zf zf zf zf zf X1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 : 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 X2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 Need Rank(n-1) applied toeach stride instead of the entire pTree. The result from stride=j gives the jth entry of SpS(X,d(x,X-x)) Parallelize over a large cluster? Ct(P&Pi): revise the Count proc to kick out count for each stride (involves loop down pTree by register-lengths? What does P represent after each step?? How does alg go on 2pDoop (w 2 level pTrees) where each stride is separate Note: using d, not d2 (fewer pTrees). Can we estimate d? (using truncated McClarin series) P3 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 0 0 0 0 0 0 P2 0 0 0 0 1 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 1 1 0 0 0 1 0 1 1 1 0 P0 0 0 1 1 0 0 0 1 0 0 0 1 1 0 1 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 : 0 1 1 1 1 0 1 1 1 1 0 1 0 0 1 1 0 0 0 0 1 1 1 1 1 1 0 0 1 0 d(xy) 0 2 1 3 4 8 14 13 14 12 12 13 13 14 9 2 0 1 2 2 6 12 11 12 10 11 12 12 13 8 : 14 13 13 11 11 8 11 9 9 7 2 1 2 0 5 9 8 8 6 6 5 11 9 9 7 3 4 4 5 0 P'3 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 : 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 1 1 1 0 0 0 1 1 1 1 1 1 P'2 1 1 1 1 0 1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 1 0 1 1 0 0 0 1 : 0 0 0 1 1 1 1 1 1 0 1 1 1 1 0 1 1 1 0 0 0 1 1 1 0 1 0 0 0 1 P'0 1 1 0 0 1 1 1 0 1 1 1 0 0 1 0 1 1 0 1 1 1 1 0 1 1 0 1 1 0 1 : 1 0 0 0 0 1 0 0 0 0 1 0 1 1 0 0 1 1 1 1 0 0 0 0 0 0 1 1 0 1 P1 0 1 0 1 0 0 1 0 1 0 0 0 0 1 0 1 0 0 1 1 1 0 1 0 1 1 0 0 0 0 : 1 0 0 1 1 0 1 0 0 1 1 0 1 0 0 0 0 0 1 1 0 1 0 0 1 1 0 0 0 0 P'1 1 0 1 0 1 1 0 1 0 1 1 1 1 0 1 0 1 1 0 0 0 1 0 1 0 0 1 1 1 1 : 0 1 1 0 0 1 0 1 1 0 0 1 0 1 1 1 1 1 0 0 1 0 1 1 0 0 1 1 1 1 23 * + 22 * + 21 * + 20 * 0 0 1 = 1 0 n=3: c=Ct(P&P3)=10< 14, p=14–10=4; P=P&P' (elim 10 val8) n=2: c=Ct(P&P2)= 1 < 4, p=4-1=3; P=P&P' (elim 1 val4) n=1: c=Ct(P&P1)=2 < 3, p=3-2=1; P=P&P' (elim 2 val2) n=0: c=Ct(P&P0 )=2>=1 P=P&P0 (elim 1 val<1) 23 * + 22 * + 21 * + 20 * 0 0 1 = 1 0 n=3: c=Ct(P&P3)=9< 14, p=14–9=5; P=P&P' (elim 9 val8) n=2: c=Ct(P&P2)= 0 < 5, p=5-0=5; P=P&P' (elim 0 val4) n=1: c=Ct(P&P1)=4 < 5, p=5-4=1; P=P&P' (elim 4 val2) n=0: c=Ct(P&P0 )=1>=1 P=P&P0 (elim 1 val<1 23 * + 22 * + 21 * + 20 * 0 0 1 = 1 0 n=3: c=Ct(P&P3)= 9 < 14, p=14–9=5; P=P&P' (elim 9 val8) n=2: c=Ct(P&P2)= 2 < 5, p=5-2=3; P=P&P' (elim 2 val4)2 n=1: c=Ct(P&P1)=2 < 3, p=3-2=1; P=P&P' (elim 2 val2) n=0: c=Ct(P&P0 )=2>=1 P=P&P0 (elim 1 val<1) 23 * + 22 * + 21 * + 20 * 0 0 1 = 3 1 n=3: c=Ct(P&P3)= 6 < 14, p=14–6=8; P=P&P' (elim 6 val8) n=2: c=Ct(P&P2)= 7 < 8, p=8-7=1; P=P&P' (elim 7 val4)2 n=1: c=Ct(P&P1)=11, p=1-1=0; P=P&P (elim 0 val2) n=0: c=Ct(P&P0 )=1 0 P=P&P0 (elim 0)

  7. ANDing Multi-Level pTrees 1. A≡AND(lev1s)= resultLev1; 2. If (Ak=0 &  operand s.t. Lev0k is pure0) resultLev0k = pure0; ElseIf (Ak =1) resultLev0k = pure1; Else resultLev0k = AND(lev0s); A C B D E 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 E1-a,f Eb-e A1-6 A7-a Ab-f C1-5,f C6-e B1-f D1-f Levels are objects w methods: AND,OR,Comp,Add,Mult, Neg.. Map Reduce terminology (ptrs="maps", methods="reducers"?) 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 B= P33P32 A= E.g., P13P12 B1-f: all identical D(L0) P33P43 C= P13P33 E(L1) P13P23 A1-6: pure0, resultLev01-6 is pure0 2pDoop:2-Level Hadoop (key-value) pTreebase. pX PXX M(1=mixed else 0)XX All level-0 pTrees in the range P33..P40 are identical (= p13..p20 respectively). Here, all are mixed. All level-0 pTrees in the range P13..P20 are pure. A7-a=1, resultLev07-a is pure1 Ab-f: pure0, resultLev0b-fis pure0 And that purity is given by p12..p20 resp. Level-1: P13 P12 P11 P10 P23 P22 P21 P20 P33 P32 P31 P30 P43 P42 P41 P40 M1* M2* M3* M4* p13 p12 p11 p10 p23 p22 p21 p20 pure0 pure1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 All pairwise ptrees put in 2pDoopupon data capture? What I'm after here is SpSX(d(x,{yX|yx}) and I would like to compute this SpS without looping on X. All 2-level pTrees for SpS[XX,(x1-x3)2+(x2-x4)2] put in 2pDoop. embarrassingly parallelizable P131-f P121-f P111-f P101-f P231-f P221-f P211-f P201-f P331-f P321-f P311-f P301-f P431-f P421-f P411-f P401-fLevel-0 ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡

  8. Level-1 key map Red=pure stride (so no Level-0) 13 12 11 10 23 22 21 20 33 32 31 30 43 42 41 40 e 2 34 0 0 0 0 f5 6 0 0 0 g7 0 0 h 0 i 8 9 a 0 0 0 0 j b c 0 0 0 k d 0 0 m 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 12 11 10 23 22 21 20 33 32 31 30 43 42 41 40 33 32 31 30 43 42 41 40 13 12 11 10 23 22 21 20 33 32 31 30 43 42 41 40 e 2 3 4 f 5 6 g 7 h i 8 9 a j b c k d m 33 32 31 30 43 42 41 40 2 3 4 5 6 7 8 9 a b c d e f g h i j k m 0 0 0 0 0 0 1 1 1 0 1 1 0 1 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 0 0 0 0 1 0 1 0 1 1 0 0 0 0 1 0 1 0 1 0 0 1 0 1 0 0 1 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 Level-0: key map (6-e) = f else pur0 (6-e) = g else pur0 (6-e) = h else pur0 (6-e) = e else pur0 In this 2pDoop KEY-VALUE DB, we list keys. Should we bitmap? Each bitmap is a pTree in the KVDB. Each of these is existing, e.g., e here 5,7-a,f=f else pur0 5,7-a,f=g else pur0 5,7-a,f=h else pur0 234789bcefh else pr0 234789bcefg els pr0 124-79c-f h else pr0 (b-f) = j else pur0 (b-f) = k else pur0 (b-f) = i else pur0 (b-f) = m else pur0 (a) = j else pur0 (a) = k else pur0 (a) = m else pur0 p23p43 p23p42 p13p33 + p13p32 + -27( =SpS(XX, (3-6,8,9) k, els pr0 (3-6,8,9) m els pr0 + p13p31 + p23p41 p13+p23+p33+p43 +p13p12+ p23p22+ p33p32 + +p43p42 ) -26( 26( 124679bd m els pr0 p23p21 + p33p31 + p43p41 ) p13p11+ -25( p13p30 +p23p40 25( +p12p31 +p22p41 +p12p32 +p22p42 p12+p22+p32+p42 +p23p20 +p13p10+ +p33p30 +p43p40 24( -24(p12p30 +p22p40 +p32p31 +p42p41 ) +p12p11+ +p22p21 p22p20 + p32p30 + p42p40 ) p12p10+ -23(p11p31 +p11p30 +p21p41 +p21p40 23( p10+p20+p30+p40) p11+p21+p31+p41 +p11p10 + +p21p20 + +p31p30 -22(p10p30 +p20p40 +p41p40 ) 22(

  9. In order to use the Rank(n-1) algorithm effectively we will need pTrees for XX and X3 X4 p32 p31 p30 p43 p42 p41 p40 IDY z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 : 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 : 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 : 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 : 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 : 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 : 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 : 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 : 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 : z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf : 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 : 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 3 2 3 6 9 15 14 15 13 10 11 9 11 7 z1 z2 z3 z4 z5 z6 z7 z8 z9 za zb zc zd ze zf 1 1 2 3 2 3 1 2 3 4 9 10 11 11 8 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 SpS(XX, d2(x1,x2),(x3,x4) ) = SpS(XX, (x1-x3)2+(x2-x4)2 ) XX x___ y___ X1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 : 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 X2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 p13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p10 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 p22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p20 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 p33 IDX z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z1 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 z2 : ze ze ze ze ze ze ze ze ze ze ze ze ze ze ze zf zf zf zf zf zf zf zf zf zf zf zf zf zf zf 0 4 2 8 17 68 196 170 200 153 145 181 164 200 85 4 0 2 4 5 40 144 122 148 109 113 145 136 164 65 : 200 164 162 128 117 68 116 90 80 53 5 1 4 0 25 85 65 61 41 40 29 113 85 89 52 10 20 13 25 0

  10. =SpS(XX, p23+p23p22 + p33+p33p32 + p43+p43p42 -2 p13p33-2p13p32 -2 p23p43-2p23p42 ) + p13+p13p12 + 26( p23p21 + p33p31 + p43p41 -2 p13p31 -2 p23p41 ) + p13p11 + 25( p12+p13p10+p12p11 + p22+p23p20+p22p21 + p42+p43p40+p42p41 -2p22p42-2p23p40-2p22p41) + p32+p33p30+p32p31 + -2p12p32-2p13p30-2p12p31 24( p22p20 + p32p30 + p42p40 -2 p12p30 -2 p22p40 ) + p12p10 + 23( -2p11p31-2p11p30 -2 p21p41-2p21p40 ) + p10 + p20 + p30 + p40 -2p10p30 -2p20p40) p11+p11p10 + p21+p21p20 + p31+p31p30 + p41+p41p40 22( p13p33+p13p32 +p23p43+p23p42 ) + =SpS(XX, -27( - p13p31 - p23p41 ) + p13+p23+p33+p43 +p13p12 + +p23p22 + +p33p32 + +p43p42 26( p23p21 + p33p31 + p43p41 p13p11 + - p12p32 - p13p30 - p12p31 - - p22p42 - p23p40 - p22p41 ) + 25( p12+p13p10+p12p11 + p22+p23p20+p22p21 + p42+p43p40+p42p41 - p12p30 - p22p40 ) + p32+p33p30+p32p31 + 24( p22p20 + p32p30 + p42p40 - ) + p12p10 + p11p31- p11p30 - p21p41- p21p40 23( p11+p11p10 + p21+p21p20 + p31+p31p30 + - p10p30 -p20p40 p41+p41p40 ) + 22( p10 + p20 + p30 + p40) p13 p12 p11 p10 p23 p22 p21 p20 p33 p32 p31 p30 p43 p42 p41 p40 * 1 1 1 1 1 1 1 * 1 1 1 1 1 * 1 1 1 * 1 * 1 1 1 1 1 1 1 * 1 1 1 1 1 * 1 1 1 * 1 * 1 1 1 * 1 1 * 1 * * 1 1 1 * 1 1 * 1 * p13 p12 p11 p10 p23 p22 p21 p20 p33 p32 p31 p30 p43 p42 p41 p40 p13p33 + p13p32 + p23p43 p23p42 -27( =SpS(XX, + )+ p13p31 + p23p41 p13+p23+p33+p43 )+ +p13p12+p23p22 + +p33p32 + +p43p42 ) -26( 26( +p23p40 p23p21 + p33p31 + p43p41 ) p13p11+ -25( p13p30 25( +p12p31 +p22p41 +p12p32 +p22p42 )+ p12+p22+p32+p42 +p23p20 +p43p40 -24(p12p30 )+ +p13p10+ +p33p30 +p22p40 24( +p32p31 +p42p41 ) +p12p11+ +p22p21 p22p20 + p32p30 + p42p40 ) p12p10+ -23(p11p31 +p11p30 +p21p41 +p21p40 23( )+ p11+p21+p31+p41 +p11p10 + +p21p20 + +p31p30 -22(p10p30+p20p40 +p41p40 ) )+ 22( p10+p20+p30+p40) SpS[ XX, d2(x=(x1,x2), y=(x3,x4)] = SpS[ XX, (x1-x3)2+(x2-x4)2 ] = SpS(XX, x1x1 + x2x2 + x3x3 + x4x4 - 2x1x3 -2x2x4) piipii=pii (no processing) Only 44 the pairwise products need computing.

More Related