1 / 40

Scaling Decision Tree Induction

Scaling Decision Tree Induction. Outline. Why do we need scaling? Cover state of the art methods Details on my research (which is one of the state of the art methods). Problems Scaling Decision Trees. Data doesn’t fit in RAM Numeric attributes require repeated sorting

marlie
Download Presentation

Scaling Decision Tree Induction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Scaling Decision Tree Induction

  2. Outline • Why do we need scaling? • Cover state of the art methods • Details on my research (which is one of the state of the art methods)

  3. Problems Scaling Decision Trees • Data doesn’t fit in RAM • Numeric attributes require repeated sorting • Noisy datasets lead to very large trees • Large datasets fundamentally different from smaller ones • Can’t store the entire dataset • Underlying phenomenon changes over time

  4. Current State-Of-The-Art • Disk based methods • Sprint • SLIQ • Sampling methods • BOAT • VFDT & CVFDT • Data Stream Methods • VFDT & CVFDT

  5. SPRINT/SLIQ • Shafer, Agrawal, Mehta • In the IBM Intelligent Miner for Data • Learns the same tree as traditional method but works with data on disk • One scan over the data per level of the induced tree

  6. SPRINT/SLIQ Details • Split the dataset into one file per attribute • (value, record ID) • Pre-sort each numeric attribute’s file • Do one scan over each file, find best split point • Use hash-tables to split the files maintaining sort order • Recur

  7. SPRINT/SLIQ Splitting Example Test Attrib To Split > val | rec val | rec 10 | 1 3 | 3 10 | 1 14 | 6 5 | 2 14 | 6 20 | 4 6 | 5 20 | 2 9 | 1 25 | 4 10 | 4 30 | 3 12 | 6 40 | 5 < ‘hashtable’ 20 | 2 1 | > 30 | 3 2 | < 40 | 5 3 | < 4 | > 5 | < 6 | >

  8. BOAT • Gehrke, Ganti, Ramakrishnan, Loh • Learns the same tree as traditional methods but can be as much as 3x faster than SPRINT/SLIQ • When things work out learns more than one level of tree in one scan over the database

  9. BOAT Details • Read a sample of data into memory • Learn N trees via traditional methods on bootstrap samples from this sample • Keep any subset of the N trees that is exactly the same • Verify the subtree with a scan over all data • When this fails revert to SPRINT/SLIQ

  10. BOAT Example x1? x1? x1? male female male female male female x2? x2? x2? > 65 <= 65 > 67 <= 67 > 61 <= 61 x3? no yes x1? male female x2? > 61 <= 67 ?

  11. VFDT/CVFDT • Hulten, Spencer, Domingos • With high probability learns what traditional methods would learn, but much faster • Learns from data stream instead of data base • CVFDT is extension to time changing concepts

  12. Motivation • Why use a data stream model? • High data rate • Essentially infinite data • Data collected in varied circumstances • Need a algorithms that are: • Constant time per example & use each example once • Incremental • Anytime • Produce results ‘equivalent’ to traditional methods

  13. Hoeffding Trees • In order to pick split attribute for a node looking at a few example may be sufficient • Given a stream of examples: • Use the first to pick the split at the root • Sort succeeding ones to the leaves • Pick best attribute there • Continue… • Leaves predict most common class

  14. How Much Data? • Make sure best attribute is better than second • That is: • Using a statistical result: Hoeffding bound • Collect data till:

  15. Hoeffding Tree Algorithm Proceedure HoeffdingTree(Stream, δ) Let HT = Tree with single leaf (root) Initialize sufficient statistics at root For each example (X, y) in Stream Sort (X, y) to leaf using HT Update sufficient statistics at leaf Compute G for each attribute If G(best) – G(2nd best) > ε, then Split leaf on best attribute For each branch Start new leaf, init sufficient statistics Return HT x1? male female y=0 x2? > 65 <= 65 y=0 y=1

  16. Properties of Hoeffding Trees • Model may contain incorrect splits, useful? • Bound the difference with infinite data tree • Chance an arbitrary example takes different path • Intuition: example on level i of tree has i chances to go through a mistaken node

  17. VFDT (Very Fast Decision Tree) • Memory management • Memory dominated by sufficient statistics • Deactivate less promising leaves when needed • Ties: • Wasteful to decide between identical attributes • Check for splits periodically • Pre-pruning (optional) • Only make splits that improve the value of G(.) • Early stop on bad attributes • Bootstrap with traditional learner • Rescan old data when time available

  18. Experiments • Compared VFDT and C4.5 (Quinlan, 1993) • Same memory limit for both (40 MB) • 100k examples for C4.5 • VFDT settings: δ = 10^-7, τ = 5% • Domains: 2 classes, 100 binary attributes • Fifteen synthetic trees 2.2k – 500k leaves • Noise from 0% to 30%

  19. Running Times • Pentium III at 500 MHz running Linux • C4.5 takes 35 seconds to read and process 100k examples; VFDT takes 47 seconds • VFDT takes 6377 seconds for 20 million examples; 5752s to read 625s to process • VFDT processes 32k examples per second (excluding I/O)

  20. Time-Changing Data Streams • Underlying concept often changes over time • Seasonal effects • Economic cycles • Etc. • Many KDD systems assume data is sample from stationary distribution • CVFDT -- Extends VFDT for time changing data streams

  21. Dealing with Time Changing Concepts • Out-of-date data misleads learner and results in larger or less accurate models • Maintain a window of the most recent examples • When new data arrives update the window and reapply the learner • Effective when window size similar to concept drift rate • Extremely inefficient!

  22. Concept adapting VFDT • Keep up to date with a window of size w • Incrementally incorporate and forget examples • Smoothly change the induced tree • Grow speculative structure • Change structure when more accurate • Incorporates new examples in constant time instead of relearning on window: O(w) time

  23. Window (Forgetting Examples) • Keep sufficient statistics at every node • Update with new & old examples • Keep an ID and only forget where needed • Quickly update leaf predictions • Periodically check for any invalid splits • Some portion due to incorrect initial splits • The rest due to changes in the data stream

  24. Alternate Sub-Trees • When new test looks better grow alternate sub-tree • Replace the old when new is more accurate • This smoothly adjusts to changing concepts Gender? Pets? College? false Hair? true true false false true

  25. CVFDT Details • Memory Requirements • When drift present, CVFDT uses fewer nodes than VFDT • Observed good results with relatively few alternate-trees • Update time • O(# attribs * # values * # classes * path length) • Independent of training set and window size!

  26. Other things • Dynamic window size • Drastic changes in the data stream • Drastic changes in the induced model • No apparent changes (learn more detail)

  27. Synthetic Experiments • Concept based on parallel hyper-planes • Aligned axis better split attribute, rotate the hyper-planes to change structure of ‘true’ tree + + - - Concept Drift + + - -

  28. Synthetic Experiments (cont.) • Compare CVFDT with VFDT • 5 million training examples • Drift inserted by periodically rotating hyper-planes • About 8% of test points change label each drift • 100,000 examples in window • 5% noise • Results sampled every 10k examples throughout the run and averaged

  29. Error Rate vs. # Attributes

  30. Tree Size vs. # Attributes

  31. Detailed View of Single Run

  32. Varying Levels of Drift

  33. Details of Adaptation

  34. Comparison With VFDT-window • CVFDT most of the accuracy gain • VFDT: 10 min • CVFDT: 46 min • VFDT-window • Est. 548 days! VFDT-Window VFDT CVFDT

  35. Application: Web Data • Trace of all web requests from UW campus • 82.8 million requests over one-week period • Goal: to predict which pages to cache • CVFDT does better for first 70% of run • VFDT’s performance improved near end • Data seems to contain drift, but more study is needed

  36. Open Issues • Continuous Attributes • Batch version of VFDT • Very Fast Post Pruning • Extending general method to other algorithms

  37. Summary • Decision trees important, need some more work to scale to today's problems • Disk based methods • About one scan per level of tree • Sampling can produce equivalent trees much faster

More Related