Conceptual Clustering

1. Conceptual Clustering • Unsupervised, spontaneous - categorizes or postulates concepts without a teacher • Conceptual clustering forms a classification tree - all initial observations in root - create new children using single attribute (not good), attribute combinations (all), information metrics, etc. - Each node is a class • Should decide quality of class partition and significance (noise) • Many models use search to discover hierarchies which fulfill some heuristic within and/or between clusters - similarity, cohesiveness, etc.

2. Cobweb • Cobweb is an incremental hill-climbing strategy with bidirectional operators - not backtrack, but could return in theory • Starts empty. Creates a full concept hierarchy (classification tree) with each leaf representing a single instance/object. You can choose how deep in the tree hierarchy you want to go for the specific application at hand • Objects described as nominal attribute-value pairs • Each created node is a probabilistic concept (a class) which stores probability of being matched (count/total), and for each attribute, probability of being on, P(a=v|C), only counts need be stored. • Arcs in tree are just connections - nodes store info across all attributes (unlike ID3, etc.)

3. Category Utility: Heuristic Measure • Tradeoff between intra-class similarity and inter-class dissimilarity - sums measures from each individual attribute • Intra-class similarity a function of P(Ai = Vij|Ck), Predictability of C given V - Larger P means if class is C, A likely to be V. Objects within a class should have similar attributes. • Inter-class dissimilarity a function of P(Ck|Ai = Vij), Predictiveness of C given V - Larger P means A=V suggests instance is member of class C rather than some other class. A is a stronger predictor of class C.

4. Category Utility Intuition • Both should be high over all (most) attributes for a good class breakdown • Predictability: P(V|C) could be high for multiple classes, giving a relatively low P(C|V), thus not good for discrimination • Predictiveness: P(C|V) could be high for a class, while P(V|C) is relatively low, due to V occurring rarely, thus good for discrimination, but not intra-class similarity • When both are high, get best categorization balance between discrimination and intra-class similarity

5. Category Utility • For each category sum predictability times predictiveness for each attribute weighted by P(Ai = Vij), with k proposed categories, i attributes, j values/attribute The expected number of attribute values one could guess given C

6. Category Utility • Category Utility is the increase in expected attributes that could be guessed, given a partitioning of categories - leaf nodes. • CU({C1, C2, ... Ck}) = • K normalizes CU for different numbers of categories in the candidate partition • Since incremental, there is a limited number of possible categorization partitions • If Ai = Vij is independent (irrelevant) of class membership, CU = 0

7. Cobweb Learning Algorithm 1. Incrementally add a new training example 2. Recurse down the at root until new node with just this example is added. Update appropriate probabilities at each level. 3. At each level of the tree calculate the scores for all valid modifications using category utility (CU) 4. Depending on which of the following gives the best score: • Classify into an existing class - then recurse • Create a new class node – done, can get next example • Combine two classes into a single class (Merging) - then recurse • Divide a class into multiple classes (Splitting) - then recurse

8. Cobweb Learning Mechanisms • Classifying (Matching) - calculate overall CU for each case of putting the example in a node at current level • New Class - calculate overall CU for putting example into a single new class- Note gradient descent (greedy) nature. Does not go back and try all possible new partitions. • If created from internal node, simply add • If created from leaf node, split into two, one for new and old • These alone are quite order dependent - splitting and merging allow bi-directionality - ability to undo

9. Cobweb Learning Mechanisms • Merging - For best matching node (the one that would be chosen for classification) and the second best matching node at that level, calculate CU when both are merged into one node, with two children • Splitting - For best matching node, calculate CU if that node were deleted and it’s children added to the current level. • Both schemes could be extended to test other nodes, at the cost of increased computational complexity • Can overcome initial “misconceptions”

10. Cobweb Comments • Generalization done by just executing recursive classification step • Could use different variations on CU and search strategy • Complexity: O(AVB2logK) for each example, where B is branching factor, A (attributes), V (average number of values), K (classes) • Empirically, B usually between 2 and 5 • Does not directly handle noise - no defined significance mechanism • Tends to make “bushy” trees, however high levels should be most important class categories (because of merge/split causing best breaks to float up, though no optimal guarantee), and one could just use nodes highest in the tree for classification • Does not support continuous values

11. Extensions - Classit • Cannot store probability counts for continuous data • Classit uses a scheme similar to Cobweb, but assumes normal distribution around an attribute and thus can just store a mean and variance - not always a reasonable assumption • Also uses a formal cut-off (significance) mechanism to better support generalization and noise handling (a class node can then include outliers) • More work needed