Scalable Data Mining

autonlab.org Scalable Data Mining Brigham Anderson, Andrew Moore, Dan Pelleg, Alex Gray, Bob Nichols, Andy Connolly The Auton Lab, Carnegie Mellon University www.autonlab.org

Outline autonlab.org • Kd-trees • Fast nearest-neighbor finding • Fast K-means clustering • Fast kernel density estimation • Large-scale galactic morphology

33 Distance Computations Nearest Neighbor - Naïve Approach • Given a query point X. • Scan through each point Y • Takes O(N) time for each query!

Speeding Up Nearest Neighbor • We can speed up the search for the nearest neighbor: • Examine nearby points first. • Ignore any points that are further then the nearest point found so far. • Do this using a KD-tree: • Tree based data structure • Recursively partitions points into axis aligned boxes.

KD-Tree Construction We start with a list of n-dimensional points.

KD-Tree Construction X>.5 YES NO We can split the points into 2 groups by choosing a dimension X and value V and separating the points into X > V and X <= V.

KD-Tree Construction X>.5 YES NO We can then consider each group separately and possibly split again (along same/different dimension).

KD-Tree Construction X>.5 YES NO Y>.1 NO YES We can then consider each group separately and possibly split again (along same/different dimension).

KD-Tree Construction We can keep splitting the points in each set to create a tree structure. Each node with no children (leaf node) contains a list of points.

KD-Tree Construction We will keep around one additional piece of information at each node. The (tight) bounds of the points at or below this node.

KD-Tree Construction Use heuristics to make splitting decisions: • Which dimension do we split along? Widest • Which value do we split at? Median of value of that split dimension for the points. • When do we stop? When there are fewer then m points left OR the box has hit some minimum width.

Nearest Neighbor with KD Trees We traverse the tree looking for the nearest neighbor of the query point.

Nearest Neighbor with KD Trees Examine nearby points first: Explore the branch of the tree that is closest to the query point first.

Nearest Neighbor with KD Trees When we reach a leaf node: compute the distance to each point in the node.

Nearest Neighbor with KD Trees Then we can backtrack and try the other branch at each node visited.

Nearest Neighbor with KD Trees Each time a new closest node is found, we can update the distance bounds.

Nearest Neighbor with KD Trees Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.

Metric Trees • Kd-trees rendered worse-than-useless in higher dimensions • Only requires metric space (a well-behaved distance function)

What does k-means do?

K-means • Ask user how many clusters they’d like. (e.g. k=5)

K-means • Ask user how many clusters they’d like. (e.g. k=5) • Randomly guess k cluster Center locations

K-means • Ask user how many clusters they’d like. (e.g. k=5) • Randomly guess k cluster Center locations • Each datapoint finds out which Center it’s closest to. (Thus each Center “owns” a set of datapoints)

K-means • Ask user how many clusters they’d like. (e.g. k=5) • Randomly guess k cluster Center locations • Each datapoint finds out which Center it’s closest to. • Each Center finds the centroid of the points it owns

K-means • Ask user how many clusters they’d like. (e.g. k=5) • Randomly guess k cluster Center locations • Each datapoint finds out which Center it’s closest to. • Each Center finds the centroid of the points it owns… • …and jumps there • …Repeat until terminated! • For basic tutorial on kmeans, see any machine learning text book , or www.cs.cmu.edu/~awm/tutorials/kmeans.html

K-means • Ask user how many clusters they’d like. (e.g. k=5) • Randomly guess k cluster Center locations • Each datapoint finds out which Center it’s closest to. • Each Center finds the centroid of the points it owns

K-means search guts I must compute Sxi of all points I own I must compute Sxi of all points I own I must compute Sxi of all points I own I must compute Sxi of all points I own

K-means search guts I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

K-means search guts I will compute Sxi of all points I own in left rectangle, then right subrectangle, then add ‘em I will compute Sxi of all points I own in left subrectangle, then right subrectangle, then add ‘em I will compute Sxi of all points I own in left rectangle, then right subrectangle, then add ‘em I will compute Sxi of all points I own in left rectangle, then right subrectangle, then add ‘em

In recursive call… I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

Find center nearest to rectangle In recursive call… I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

Find center nearest to rectangle • For each other center: can it own any points in rectangle? In recursive call… I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

Find center nearest to rectangle • For each other center: can it own any points in rectangle? • If not, PRUNE Pruning I will just grab Sxi from the cached value in the node I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

Find center nearest to rectangle • For each other center: can it own any points in rectangle? • If yes, recurse... Pruning not possible at previous level… I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle But.. maybe there’s some optimization possible anyway? I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

Find center nearest to rectangle • For each other center: can it own any points in rectangle? • If yes, recurse... Blacklisting I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle But.. maybe there’s some optimization possible anyway? A hopeless center never needs to be considered in any recursion I will compute Sxi of all points I own in rectangle I will compute Sxi of all points I own in rectangle

Example • Example generated by: • Dan Pelleg and Andrew Moore. Accelerating Exact k-means Algorithms with Geometric Reasoning. Proc. Conference on Knowledge Discovery in Databases 1999, (KDD99) (available on www.autonlab.org)

K-means continues…

Scalable Data Mining

Scalable Data Mining

Presentation Transcript

Data Mining: Data

Data Mining: Data

Scalable data mining for functional genomics and metagenomics

Data Mining: Data

Deterministic Annealing and Robust Scalable Data mining for the Data Deluge

Scalable data mining for functional genomics and metagenomics

Scalable Mining on Information Networks

PARSYMONY: Scalable Parallel Data Mining

SLIQ: A Fast Scalable Classifier for Data Mining

SPRINT: A Scalable Parallel Classifier for Data Mining

Scalable Benchmarks and Kernels for Data Mining and Analytics

Data Mining: Data

SPRINT : A Scalable Parallel Classifier for Data Mining

Data Mining: Data

Towards Scalable Critical Alert Mining

Scalable data mining for functional genomics and metagenomics

SLIQ: A Fast Scalable Classifier for Data Mining

Scalable Algorithms for Association Mining

Data Mining: Data