1 / 11

CS 9633 Machine Learning Feature Selection

CS 9633 Machine Learning Feature Selection. References: W. M. Weiss and C. A. Kulikowski, Computer Systems that Learn, 1991, Morgan Kaufmann. R. Kohavi, and G. John, Wrappers for Feature Subset Selection, 1998, Artificial Intelligence. Feature Selection.

lexi
Download Presentation

CS 9633 Machine Learning Feature Selection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 9633 Machine LearningFeature Selection References: W. M. Weiss and C. A. Kulikowski, Computer Systems that Learn, 1991, Morgan Kaufmann. R. Kohavi, and G. John, Wrappers for Feature Subset Selection, 1998, Artificial Intelligence Computer Science Department CS 9633 KDD

  2. Feature Selection • Some features are more informative than others • Some learning methods use all of the features (k-nn) and are quite sensitive to correlated attributes or to noise (uninformative attributes). • Some learning algorithms include feature selection as part of functionality Computer Science Department CS 9633 KDD

  3. Feature Selection as Search Computer Science Department CS 9633 KDD

  4. Search strategies • Starting point • Forward selection • Backward elimination • Organization of search • Exhaustive • Greedy • Stepwise selection or elimination (add or remove features at each point) • Strategy for selecting among alternative subsets Computer Science Department CS 9633 KDD

  5. Strategy for Selection • Feature filter • features are selected independent of the learning algorithm • Wrapper approach • Generate set of candidate features • Run induction algorithm • Measure performance with the learning algorithm to evaluate feature set • Accuracy with training set • Cross validation Computer Science Department CS 9633 KDD

  6. Filter Approach Input Features Feature subset selection Induction algorithm Computer Science Department CS 9633 KDD

  7. Wrapper Approach Test set Feature selection search Induction algorithm Training set Feature set Performance estimation Feature set Feature evaluation Feature set Hypothesis Induction algorithm Test set Estimated Accuracy Final Evaluation Computer Science Department CS 9633 KDD

  8. Filter versus Wrapper • Wrapper approach finds best features for a specific learning algorithm • Wrapper approach generally has much higher computational cost Computer Science Department CS 9633 KDD

  9. Criterion for Halting Search • Filter approach • Rank features by usefulness score and determine breakpoint • When each combination of values for the selected attributes maps onto a single class value • Wrapper approach • Stop adding or removing when performance does not increase • Revise as long as performance does not degrade • Keep adding and removing until reach the other end of the search space and pick the best Computer Science Department CS 9633 KDD

  10. Measures for filter methods • Pearson correlation coefficient (2 class problems) • Residual sum of squares • Adjusted R-square • Minimum mean residual Computer Science Department CS 9633 KDD

  11. Search strategies for wrapper approach • Forward greedy • Backward greedy • Beam search • Branch and bound • Genetic algorithm Computer Science Department CS 9633 KDD

More Related