Converting Categories to Numbers for Approximate Nearest Neighbor Search

1. Converting Categories to Numbers for Approximate Nearest Neighbor Search 嘉義大學資工系 郭煌政 2004/10/20

2. Outline • Introduction • Motivation • Measurement • Algorithms • Experiments • Conclusion

3. Introduction • Memory-Based Reasoning • Case-Based Reasoning • Instance-Based Learning • Given a training dataset and a new object, predict the class (target value) of the new object. • Focus on table data

4. Introduction • K Nearest Neighbor Search • Compute similarity between the new object and each object in the training dataset. • Linear time to the size of the dataset • Similarity: Euclidean distance • Multi-dimension Index • Spatial data structure, such as R-tree • Numeric data

5. Introduction • Indexing on Categorical Data? • Linear order of the categories • Existing correct ordering? • Best ordering? • Store the mapped data on a multi-dimensional data structure as filtering mechanism

6. Measurement for Ordering Ordering Problem Given an undirected weighted complete graph, a simple path is an ordering of the vertices. The edges are the distances between pairs of vertices. The ordering problem is to find a path, called ordering path, of maximal value according to a certain scoring function.

7. Measurement for Ordering • Relationship ScoringReasonable Ordering Score • In an ordering path <v1, v2, …, vn>, 3-tuple <vi-1, vi, vi+1> is reasonable if and only if dist(vi-1, vi+1) ≧dist(vi-1, vi) anddist(vi-1, vi+1) ≧dist(vi, vi+1).

8. Measurement for Mapping • Pairwise Difference Scoring • Normalized distance matrix • Mapping values of categories • Distm(vi, vj) = |mapping(vi) - mapping(vj)|

9. Algorithms • Prim-like Ordering • Kruskal-like Ordering • Divisive Ordering • GA Approach Ordering • A vertex is a category • A graph represent a distance matrix

10. Prim-like Ordering Algorithm • Prim’s Minimum Spanning Tree • Initially, choose a least edge (u, v) • Add the edge to the tree; S = {u, v} • Choose a least edge connecting a vertex in S and a vertex, w, not in S • Add the edge to the tree; Add w to S • Repeat until all vertices are in S

11. Prim-like Ordering Algorithm • Prim-like Ordering • Choose a least edge (u, v) • Add the edge to the ordering path; S = {u, v} • Choose a least edge connecting a vertex in S and a vertex, w, not in S • If the edge creates a circle on the path, discard the edge, and choose again • Else, add the edge to the ordering path; Add w to S • Repeat until all vertices are in S

12. Kruskal-like Ordering Algorithm • Kruskal Minimum Spanning Tree • Initially, choose a least edge (u, v) • Add the edge to the tree; S = {u, v} • Choose a least edge as long as the edge does not create a circle in the tree • Add the edge to the tree; Add the two vertiecs to S • Repeat until all vertices are in S

13. Kruskal-like Ordering Algorithm • Kruskal-like Ordering • Initially, choose a least edge (u, v) and add it to the ordering path; S = {u, v} • Choose a least edge as long as the edge does not create a circle in the tree, anddegree of either vertex on the path is <= 2 • Add the edge to the ordering path; Add the two vertices to S • Repeat until all vertices are in S • Heap array can be used to speed up choosing least edge

14. Divisive Ordering Algorithm • Idea: • Pick a central vertex, and split the rest vertices • Building a binary tree: vertices are the leaves • Central Vertex:

15. P B A AL BL AR BR Divisive Ordering Algorithm • AR is closer to P than AL is. • BL is closer to P than BR is.

16. Clustering • Splitting a Set of Vertices into Two Groups • Each group has at least one vertex • Close (similar) vertices in same groupDistant vertices in different groups • Clustering Algorithms • Two clusters

17. Clustering • Clustering • Grouping a set of objects into classes of similar objects • Agglomerative Hierarchical Clustering Algorithm • Singleton clusters • Merge similar clusters

18. Clustering • Clustering Algorithm: Cluster Similarity • Single linkdist(Ci, Cj) = min(dist(p, q)), p in Ci, q in Cj • Complete linkdist(Ci, Cj) = max(dist(p, q)), p in Ci, q in Cj • Average link -- adopted in our studydist(Ci, Cj) = avg(dist(p, q)), p in Ci, q in Cj • others

19. Clustering • Clustering Implementation Issues • Which pair of clusters to be merged:Keep cluster-to-cluster similarity for each pair • Recursively partition sets of vertices while building the binary tree:Non-recursive version with a stack

20. GA Ordering Algorithm • Genetic Algorithm for Optimal Problems • Chromosome: solution • Population: pool of solutions • Genetic Operations • Crossover • Mutation

21. GA Ordering Algorithm • Encoding a Solution • Binary string • Ordered list of categories – in our ordering problem • Fitness Function • Reasonable ordering score • Selecting Chromosomes for crossover • High fitness value => high probability

22. GA Ordering Algorithm • Crossover • Single point • Multiple points • Mask • Crossover AB | CDE and BD | AEC • Results in ABAEC and BDCDE => Illegal

23. GA Ordering Algorithm • Repair Illegal Chromosome ABAEC • AB*EC => fill D in * position • Repair Illegal Chromosome ABABC • AB**C • D and E are missing • Which one is closest to B, fill it in first * position

24. Mapping Function • Ordering Path <v1, v2, …, vn> • Mapping(vi) =

25. Experiments • Synthetic Data (width/length = 5)

26. Experiments • Synthetic Data (width/length = 10)

27. Experiments • Synthetic Data:Reasonable Ordering Score for Divisive Algorithm • width/length = 5 => 0.82 • width/length = 10 => 0.9 • No Ordering => 1/3 • Divisive algorithm is better than Prim-like algorithm when number of categories > 100

28. Experiments • Synthetic Data (width/length = 5)

29. Experiments • Synthetic Data (width/length = 10)

30. Experiments • Divisive Ordering is best among the three ordering algorithms • For divisive ordering algorithm on > 100 categories, RMSE scores are around 0.07 when width/length = 10, and are around 0.05 when width/length = 10. • Prim-like ordering algorithm: 0.12 and 0.1, respectively.

31. Experiments • “Census-Income” dataset from the University of California, Irvine (UCI) KDD Archive • 33 nominal attributes, 7 continuous attributes • Sample 5000 records for training dataset. • Sample 2000 records for approximate KNN search experiment.

32. Experiments • Distance Matrix: distance between two categories • V. Ganti, J. Gehrke, and R. Ramakrishnan, “CACTUS-Clustering Categorical Data Using Summaries,” ACM KDD, 1999 • D = {d1, d2, …, dn} of n tuples. • D is subset of D1 * D2 * … * Dk, where Di is a categorical domain, for 1 ≦ i ≦ k. • di = <ci1, ci2, …, cik>.

33. Experiments

34. Experiments • Approximate KNN – nominal attributes

35. Experiments • Approximate KNN – nominal attributes

36. Experiments • Approximate KNN – nominal attributes

37. Experiments • Approximate KNN – all attributes

38. Experiments • Approximate KNN – all attributes

39. Experiments • Approximate KNN – all attributes

40. Conclusion • Developed Ordering Algorithms • Prim-like • Krusal-like • Divisive • GA-based • Devised Measurement • Reasonable ordering score • Root mean squared error

41. Conclusion • What next? • New categories, new mapping function • New index structure? • Training mapping function for a given ordering path.

42. Thank you.