1 / 27

Improving Data Mining Utility with Projective Sampling

This research explores the use of projective sampling to optimize data mining utility by estimating the optimal training set size. It discusses learning curves, progressive sampling strategies, and empirical results.

jhurley
Download Presentation

Improving Data Mining Utility with Projective Sampling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving Data Mining Utility with Projective Sampling Mark Last Department of Information Systems Engineering Ben-Gurion University of the Negev, Beer-Sheva, Israel E-mail: mlast@bgu.ac.il Home Page: http://www.bgu.ac.il/~mlast/ Mark Last (mlast@bgu.ac.il)

  2. Agenda • Introduction • Learning Curves and Progressive Sampling • The Projective Sampling Strategy • Empirical Results • Conclusions and Future Research Mark Last (mlast@bgu.ac.il)

  3. Motivation: Data is not “born” free • The training data is often scarce and costly • Real-world examples • A limited number of patient records stored by a hospital • Results of a costly engineering experiment • Seasonal records in an agricultural database • Even when the raw data is free, its preparation may still be labor intensive! • Critical question • Should we spend our resources (time and/or money) on acquiring more examples? Mark Last (mlast@bgu.ac.il)

  4. Total Cost of the Classification Process (based on Weiss and Tian, 2008) Training Set Score Set • Total Cost = n·Ctr + err(n)·|S|·Cerr + CPU(n)·Ctime • Ctr - cost of acquiring and labeling each new training example • Cerr - cost of each misclassified example from the score set • Ctime– cost per one unit of CPU time • n –number of training set examples used to induce the model • S - the score setof future examples to be classified by the model • err (n) – the model error rate measured on the score set • CPU(n) – CPU time required to induce the model Used to induce the classification model Future examples to be classified by the model Mark Last (mlast@bgu.ac.il)

  5. What is this research about? • Problem Statement • Find the best training set size n* that is expected to maximize the overall utility (minimize the Total Cost) • Basic Idea - Projective Sampling • Estimate the optimal training set size using learning and run-time curves projectedfrom a small subset of potentially available data • Research Objectives • Calculate the optimal training set size for a variety of learning curve equations (with and without CPU costs) • Improve the utility of the data mining process using the best fitting curves for a given dataset and an algorithm Mark Last (mlast@bgu.ac.il)

  6. Some Learning Curves for a Decision-Tree Algorithm Slow rise Rapid rise with oscillations Rapid rise Plateau Rapid rise Slow rise Mark Last (mlast@bgu.ac.il)

  7. The Best Fit for a Learning Curve • Frey and Fisher (1999) • The power law is the best fit for modeling the C4.5 error rates • Last (2007) • The power law is the best fit for modeling the error rates of an oblivious decision-tree algorithm (Information Network) • Singh (2005) • The power law is only second best to the logarithmic regression for ID3, k-Nearest Neighbors, Support Vector Machines, and Artificial Neural Networks Mark Last (mlast@bgu.ac.il)

  8. Progressive Sampling Strategy(Provost et al., 1999, Weiss and Tian, 2008) • General strategy • Start with some initial amount of training data n0 • Iteratively increase the training set until there is an increase in total cost • Popular schedules • Uniform (arithmetic) sampling • n0, n0+ , n0+ 2 ,… • Geometric Sampling • n0, a∙n0,a2∙n0,… Mark Last (mlast@bgu.ac.il)

  9. Limitations of Progressive Sampling • Overfitting some local perturbations in the error rate • Progressive sampling costs may exceed the optimal ones by 10%-200% (Weiss and Tian, 2008) • Potential overhead associated with purchasing and pre-processing each sampling increment (especially with uniform sampling). • Our expectation • The projective sampling strategy should reduce data mining costs by estimating the optimal training set size from a small subset of potentially available data Mark Last (mlast@bgu.ac.il)

  10. The Projective Sampling Strategy • Set a fixed sampling increment  • Each acquired sample = one data point • Do • Acquire a new data point • Compute Pearson's correlation coefficient for each candidate fitting function (given at least three data points) • Dependent variable: err(n) • Independent variable: training set size n • Find a function with a minimal correlation coefficient Best_Corr • Why minimal • While ((Best_Corr ≥ 0) and (n < nmax)) • Estimate the regression coefficients of the selected function • Estimate the optimal training set size n* • Induce the classification model M (n*) from n* examples Mark Last (mlast@bgu.ac.il)

  11. Candidate Fitting Functions • Learning Curves • Logarithmic : errLog (n) = a + b logn • Weiss and Tian: errWT (n) = a + bn / (n + 1) • Power Law : errPL (n) = a·nb • Exponential : errExp (n) = abn • Run-time Curves • Linear: CPUL (n) = dn • Power law: CPUPL (n) = c·nd Mark Last (mlast@bgu.ac.il)

  12. Converting Learning Curves into the Linear Form y = a’ + b’x Mark Last (mlast@bgu.ac.il)

  13. Pearson's Correlation Coefficient k – number of data points Mark Last (mlast@bgu.ac.il)

  14. Linear Regression Coefficientsy = a + bx • The least squares estimate of the slope: • The least squares estimate of the intercept k – number of data points Mark Last (mlast@bgu.ac.il)

  15. Total Cost Functions • Total CostLog (n) = n·Ctr + d∙n∙Ctime + |S|·Cerr · (a + b logn) • Total CostWT (n) = n·Ctr + d∙n∙Ctime + |S|·Cerr·( a+bn / (n+1)) • Total CostPL (n) = n·Ctr + d∙n∙Ctime + |S|·Cerr ·anb • Total CostExp (n) = n·Ctr + d∙n∙Ctime + |S|·Cerr ·abn Mark Last (mlast@bgu.ac.il)

  16. Optimizing the Training Set Size • Let • R = Cerr / Ctr • Ctr = 1 • CPUL (n) = dn • Logarithmic: • Total CostLog (n) = n + d∙n∙Ctime + |S|·R · (a + b logn) • Weiss and Tian: • Total CostWT (n) = n + d∙n∙Ctime + |S|·R ·( a+bn / (n+1)) • Power Law: • Total CostPL (n) = n + d∙n∙Ctime + |S|·R ·anb • Exponential: • Total CostExp (n) = n + d∙n∙Ctime + |S|·R ·abn Mark Last (mlast@bgu.ac.il)

  17. Experimental Settings • Ten benchmark datasets (see next slide) • Each dataset was randomly partitioned into 25%-50% of test examples and 50%-75% of examples potentially available for training . • The sampling increment  was set to 1% of the maximum possible training set size • The error rate of each increment was averaged over 10 random partitions of the training set. • Sampling schedules: Uniform, Geometric (a=2), Straw Man, Projective, Optimal • Cost Ratios (R): 1 – 50,000 • CPU Factors: 0 and 1 (per one millisecond of CPU time) Mark Last (mlast@bgu.ac.il)

  18. Datasets Description Mark Last (mlast@bgu.ac.il)

  19. Projected Fitting Functions Mark Last (mlast@bgu.ac.il)

  20. Projected and Actual Learning Curves – Small Datasets Mark Last (mlast@bgu.ac.il)

  21. Projected and Actual Learning Curves – Medium and Large Datasets Mark Last (mlast@bgu.ac.il)

  22. Comparison of Sampling Schedules R = Cerr / Ctr Mark Last (mlast@bgu.ac.il)

  23. Detailed Sampling Schedules without Induction CostsSmall Datasets Uniform Geometric, Straw Man, Projected, Optimal Mark Last (mlast@bgu.ac.il)

  24. Detailed Sampling Schedules without Induction Costs Medium and Large Datasets Geometric Uniform, Straw Man, Projected, Optimal Geometric Optimal Uniform, Straw Man, Projected Mark Last (mlast@bgu.ac.il)

  25. Conclusions • The projective sampling strategy estimates the optimal training set size by fitting an analytical function to a partial learning curve • The proposed methodology was evaluated on 10 benchmark datasets of variable size using a decision-tree algorithm. • The results show that under negligible induction costs and high data acquisition costs, the projective sampling outperforms, on average, the alternative, progressive sampling techniques. Mark Last (mlast@bgu.ac.il)

  26. Future Research • Further optimization of projective sampling schedules, especially under substantial CPU costs • Improving utility of cost-sensitive data mining algorithms • Modeling learning curves for non-random (“active”) sampling and labeling techniques Mark Last (mlast@bgu.ac.il)

  27. Thank you! Merci Beaucoup! Mark Last (mlast@bgu.ac.il)

More Related