SLIQ: A Fast Scalable Classifier for Data Mining

1. SLIQ: A Fast Scalable Classifier for Data Mining Manish Mehta, Rakesh Agrawal, Jorma Rissanen 1996. Presentation by: Vladan Radosavljevic

2. Outline • Introduction • Motivation • SLIQ Algorithm • Building tree • Pruning • Example • Results • Conclusion

3. Introduction • Most of the classification algorithms are designed for memory resident data – limited suitability for mining large datasets • Solution – build a scalable classifier - SLIQ • SLIQ – Supervised Learning in Quest, Quest was the data mining project at the IBM

4. Motivation • Recall (ID3, C4.5, CART):

5. Motivation • NON SCALABLE DECISION TREES: • The complexity lies in determining the best split for each attribute • The cost of evaluating splits for numerical attributes is dominated by the cost of sorting values at each node • The cost of evaluating splits for categorical attributes is dominated by the cost of searching for the best subset • Pruning • crossvalidation inapplicable for large datasets • divide data in two parts - training and test set - sizes, distribution???

6. Motivation • Improve scalability of tree classifiers • Previous proposals: • Sampling data at each node • Discretization of numerical attributes • Partitioning input data and build tree for each partition • All methods achieve low accuracy! • SLIQ – improve learning time without loss in accuracy!

7. SLIQ • Key features: • Tree classifier, handling both numerical and categorical attributes • Presort numerical attributes before tree has been built • Breadth first growing strategy • Goodness test – Gini index • Inexpensive tree pruning algorithm based on Minimum Description Length (MDL)

8. SLIQ - Algorithm • Eliminate the need to sort the data at each node • Create sorted list for each numerical attribute • Create class list

9. SLIQ - Algorithm • Example:

10. SLIQ - Algorithm • Split evaluation:

11. SLIQ - Algorithm • Example:

12. SLIQ - Algorithm • Update class list:

13. SLIQ - Algorithm • Example:

14. SLIQ - Algorithm • For large-cardinality categorical attributes (determined based on threshold) the best split is computed in greedy way, otherwise all possible splits are evaluated • When node becomes pure stop splitting it, then condense attribute lists by discarding examples that correspond to the pure node • SLIQ is able to scale for large datasets with no loss in accuracy – the splits evaluated with or without pre-sorting are identical

15. SLIQ - Pruning • Post pruning algorithm based on Minimum Description Length principle • Find a model that minimizes: Cost(M,D) = Cost(D|M) + Cost(M) Cost(M) - cost of the model Cost(D|M) - cost of encoding the data D if model M is given

16. SLIQ - Pruning • Cost of the data: classification error • Cost of the model: • Encoding the tree: number of bits • Encoding the splits: • numerical attribute - constant (empirically 1) • categorical attribute - depends on cardinality • The MDL pruning evaluate the code length at each node to determine whether to prune one or both child or leave the node intact

17. SLIQ - pruning • Three pruning strategies: • Full – pruning both children and convert node to the leaf • Partial – prune into the leaf or prune the left child or prune the right child or leave node intact • Hybrid – apply Full method and then partial (prune left, prune right or leave intact)

18. Results • SLIQ was tested on the datasets:

19. Results • Pruning strategy comparison:

20. Results • Accuracy:

21. Results • Scalability:

22. Conclusion • SLIQ demonstrates to be a fast, low-cost and scalable classifier that builds accurate trees • Based on empirical test which compared SLIQ to other tree based classifiers, SLIQ achieves a comparable accuracy while producing smaller decision trees • Scalability??? Memory problem when increasing number of attributes or number of classes

23. References [1] M. Mehta, R. Agrawal and J. Rissanen, "SLIQ: A Fast Scalable Classifier for Data Mining", in Proceedings of the 5th International Conference on Extending Database Technology, Avignon, France, Mar. 1996.

24. THANK YOU!