1 / 31

Randomized Algorithms

Randomized Algorithms. William Cohen. Outline. SGD with the hash trick (review) Background on o ther randomized algorithms Bloom filters Locality sensitive hashing. Learning as optimization for regularized logistic regression. Algorithm:

georgia-duran
Download Presentation

Randomized Algorithms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Randomized Algorithms William Cohen

  2. Outline • SGD with the hash trick (review) • Background on other randomized algorithms • Bloom filters • Locality sensitive hashing

  3. Learning as optimization for regularized logistic regression • Algorithm: • Initialize arrays W, A of size R andset k=0 • For each iteration t=1,…T • For each example (xi,yi) • Let Vbe hash table so that • pi = … ; k++ • For each hash value h: V[h]>0: • W[h] *= (1 - λ2μ)k-A[j] • W[h] =W[h] + λ(yi- pi)V[h] • A[j] = k

  4. Learning as optimization for regularized logistic regression • Initialize arrays W, A of size R andset k=0 • For each iteration t=1,…T • For each example (xi,yi) • k++; let V be a new hash table; let tmp=0 • For each j: xi j >0: V[hash(j)%R] += xi j • Let ip=0 • For each h: V[h]>0: • W[h] *= (1 - λ2μ)k-A[j] • ip+= V[h]*W[h] • A[h] = k • p = 1/(1+exp(-ip)) • For each h: V[h]>0: • W[h] =W[h] + λ(yi- pi)V[h] regularize W[h]’s

  5. An example 2^26 entries = 1 Gb @ 8bytes/weight

  6. Results

  7. A variant of feature hashing • Hash each feature multiple times with different hash functions • Now, each w has k chances to not collide with another useful w’ • An easy way to get multiple hash functions • Generate some random strings s1,…,sL • Let the k-th hash function for w be the ordinary hash of concatenation wsk

  8. A variant of feature hashing • Why would this work? • Claim: with 100,000 features and 100,000,000 buckets: • k=1 Pr(any duplication) ≈1 • k=2 Pr(any duplication)≈0.4 • k=3  Pr(any duplication)≈0.01

  9. Hash Trick - Insights • Save memory: don’t store hash keys • Allow collisions • even though it distorts your data some • Let the learner (downstream) take up the slack • Here’s another famous trick that exploits these insights….

  10. Bloom filter interface • Interface to a Bloom filter • BloomFilter(intmaxSize, double p); • void bf.add(Strings); // insert s • boolbd.contains(Strings); • // If s was added return true; • // else with probability at least 1-p return false; • // else with probability at most p return true; • I.e., a noisy “set” where you can test membership (and that’s it)

  11. Bloom filters • Another implementation • Allocate M bits, bit[0]…,bit[1-M] • Pick K hash functions hash(1,s),hash(2,s),…. • E.g: hash(s,i) = hash(s+ randomString[i]) • To add string s: • For i=1 to k, set bit[hash(i,s)] = 1 • To check contains(s): • For i=1 to k, test bit[hash(i,s)] • Return “true” if they’re all set; otherwise, return “false” • We’ll discuss how to set M and K soon, but for now: • Let M = 1.5*maxSize// less than two bits per item! • Let K = 2*log(1/p) // about right with this M

  12. Bloom filters • Analysis: • Assume hash(i,s) is a random function • Look at Pr(bit j is unset after n add’s): • … and Pr(collision): • …. fix m and n and minimize k: k =

  13. Bloom filters • Analysis: • Plug optimal k=m/n*ln(2) back into Pr(collision): • Now we can fix any two of p, n, m and solve for the 3rd: • E.g., the value for m in terms of n and p: p =

  14. Bloom filters: demo

  15. Bloom filters • An example application • Finding items in “sharded” data • Easy if you know the sharding rule • Harder if you don’t (like Google n-grams) • Simple idea: • Build a BF of the contents of each shard • To look for key, load in the BF’s one by one, and search only the shards that probably contain key • Analysis: you won’t miss anything…but, you might look in some extra shards • You’ll hit O(1) extra shards if you set p=1/#shards

  16. Bloom filters • An example application • discarding singleton features from a classifier • Scan through data once and check each w: • if bf1.contains(w): bf2.add(w) • else bf1.add(w) • Now: • bf1.contains(w)  w appears >= once • bf2.contains(w)  w appears >= 2x • Then train, ignoring words not in bf2

  17. Bloom filters • An example application • discarding rare features from a classifier • seldom hurts much, can speed up experiments • Scan through data once and check each w: • if bf1.contains(w): • if bf2.contains(w): bf3.add(w) • else bf2.add(w) • else bf1.add(w) • Now: • bf2.contains(w)  w appears >= 2x • bf3.contains(w)  w appears >= 3x • Then train, ignoring words not in bf3

  18. Bloom filters • More on Thursday….

  19. LSH: key ideas • Bloom filter: • Set represented by a small bit vector • Can answer containment queries fairly accurately • Locality sensitive hashing: • map feature vector xto bit vectorbx • ensure that bxpreserves “similarity” of the original vectors

  20. Random Projections

  21. Random projections u - + + - - + + - - + - + + - + - - + -u 2γ

  22. Random projections Any other direction will keep the distant points distant. u - + - + - + + - - + + - - + + - - + So if I pick a random r and r.xand r.x’ are closer than γ then probably x and x’ were close to start with. -u 2γ

  23. Random projections To make those points “close” we need to project to a direction orthogonal to the line between them u + - + - + - + - - + - + + - + - - + -u 2γ

  24. Random projections Put this another way: when is r.x>0 and r.x’<0 ? u + - + - + - - + + - - + + - + - - + -u

  25. Random projections some where in here is ok r.x’ < 0 u + - - + + - + - - + r.x > 0 + - + - + - - + -u

  26. Random projections r.x’ < 0 u + - + - - + - + + - r.x > 0 + - + - + - + - -u some where in here is ok

  27. Random projections Claim: r.x’ < 0 u + - - + + - - + + - r.x > 0 + - - + + - + - -u some where in here is ok

  28. Some math Pick random vector r, define Claim: So: And:

  29. LSH: key ideas • Goal: • map feature vector xto bit vectorbx • ensure that bxpreserves “similarity” • Basic idea: use random projections of x • Repeat many times: • Pick a random hyperplaner • Compute the inner product or r with x • Record if x is “close to” r (r.x>=0) • the next bit in bx • Theory says that is x’ and x have small cosine distance then bxand bx’ will have small Hamming distance • Famous use: near-duplicate web page detection in AltaVista (Broder, 97)

  30. LSH: key ideas • Naïve algorithm: • Initialization: • For i=1 to outputBits: • For each feature f: • Draw r(f,i) ~ Normal(0,1) • Given an instance x • For i=1 to outputBits: LSH[i] = sum(x[f]*r[i,f] for f with non-zero weight in x) > 0 ? 1 : 0 • Return the bit-vector LSH • Problem: • you need many r’s to be accurate • storing these is expensive • Ben will give us more ideas Thursday

  31. Finding neighbors of LSH vectors • After hashing x1,…,xnwe have bx1, … bxn • How do I find closest neighbors of bxi? • One approach (Indyk & Motwana): • Pick random permutation p of bits • Permute to bx1, … bxnget bx1p, … bxnp • Sort them • Check B closest neighbors of bxipin that sorted list, and keep the k closest • Repeat many times • Each pass finds some close neighbors of every element • So, can do single-link-clustering with sequential scans

More Related