A Comparative Study on Variable Selection for Nonlinear Classifiers

1. A Comparative Study on Variable Selection for Nonlinear Classifiers C. Lu1, T. Van Gestel1, J. A. K. Suykens1, S. Van Huffel1, I. Vergote2, D. Timmerman2 1Department of Electrical Engineering, Katholieke Universiteit Leuven, Leuven, Belgium, 2Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium Email address: chuan.lu@esat.kuleuven.ac.be

2. Pattern recognition: feature extraction -> classification • 1. Introduction • Variable selection refers to the problem of selecting input variables that are relevant for a given task. In pattern recognition, variable selection can have an impact on the economics of data acquisition and on the accuracy and complexity of the classifiers. • This study aims at input variable selection for nonlinear blackbox classifiers, particularly multi-layer perceptrons (MLP) and least squares support vector machines (LS-SVMs). 2. Feature extraction • Variable measure • Correlation • Mutual information (MI) • Evidence (or Bayes factor) in Bayesian framework • Classification performance • Sensitivity analysis: change in the objective function J by removing variable i: DJ(i) • Statistical partial F test (Chi-square value) • Variable Selection • Variable (feature) measure • Heuristic search: forward, backward, stepwise, hill-climbing, branch and bound… • Filter approach: filter out irrelevant attributes before induction occurs • Wrapper approaches : focus on finding attributes that are useful for performance for a specific type of model, rather than necessarily finding the relevant ones. • Feature Extraction • Feature selection or variable selection • Feature transformation • e.g. PCA, not desirable for maintaining data, difficulty in interpretation, and not immune from distortion under transformation

3. 3. Considered nonlinear classifiers: MLPs and LS-SVMs MLP Classifiers LS-SVM Classifier Bayesian Evidence Framework Inferences are divided into distinct levels. solved in dual space Note: by integrating the MLP(Mackay 1992) or LS-SVM (VanGestel, Suykens 2002) with the Bayesian evidence framework, the tuning of hyperparameters and computation of posterior class probabilities can be done in a unified way. Variable selection can also be done based on the model evidence. Model evidence

4. 4. Considered variable selection methods

5. 1 0 1 5. Experimental results on benchmark data sets I. Synthetic data: noisy XOR problem linearly inseparable. 50 random generated input data, X1, X2: {0,1} random Y: XOR(x1, x2). X3, X4: noise~N(0, 0.3) was added to X1 X2 X5, X6: noise~N(0, 0.5) was added to X1 X2 X7~X16: noise~N(0, 2). • II. Biomedical real life data set • Gene selection for leukemia classification [] • #variables: 7129, Classes: ALL, AML, • #Training data: 38; #test data: 34 Table 2. Accuracy on Test set with different number of variables *linear kernels are used for lssvm-RFE and lssvmB-FFS . Notes: Linear classifier and selection method can’t solve the XOR problem which is nonlinear. MIFSU: entropy for the first 2 binary variable smaller than the other continuous variables Bayesian LSSVM FFS: evidence for the first 2 binary variables is smaller than other continuous variables; however backward Bayesian LSSVM can always remove the other noisy variables. Table 1. LssvmRFE (using a polynomial kernel with degree 2) selected correctly the top2 variables 25 times from the 30 random trials based on the 50 noisy training data; Averaged performance on a test set of 100 examples over 30 random trials using. - MLP has 1 hidden layer with 2 hidden neurons, using Baysian MLP to determine the regularization parameter. - the LSSVM classifier uses a polynomial kernel with degree 2.

6. (2) Ovarian tumor classification # variables 27, classes: benign, malignant # training data: 265, #test data: 160 6. Conclusions • Good variable selection can improve the performance of the classifiers both in accuracy and computation. • LSSVM-RFE can be suitable for both linear and nonlinear classification problems. And can deal with the very high dimensional data. • Bayesian LSSVM forward selection can identify the important variables in some cases, however should be used with more care in the satisfaction of the assumptions. • A strategy which combines variable ranking and the wrapper methods should give more confidence in the selected variables. Table 3. Accuracy on test set with different number of variables • References • C. Lu, T. Van Gestel, et al. Preoperative prediction of malignancy of ovarian tumors using Least Squares Support Vector Machines (2002), submitted paper. • D. Timmerman, H. Verrelst, et al., Artificial neural network models for the preoperative discrimination between malignant and benign adnexal masses. Ultrasound Obstet Gynecol (1999). • J.A.K. Suykens, J. Vandewalle, Least Squares support vector machine classifiers, Neural Processing Letters (1999), 9(3). • T. Van Gestel, J.A.K. Suykens, et al., Bayesian framework for least squares support vector machine classifiers, Neural Computation (2002), 15(5). • D.J.C. MacKay, The evidence framework applied to classification networks, Neural Computation (1992), 4(5). • R. Kohavi and G. John, Wrappers for feature subset selection, Artificial intelligence, special issue on relevance 97 (1-2):273-324. • I. Guyon, J. Weston, et al. Gene selection for cancer classification using support vector machines, Machine learning (2000). • N. Kwak and C.H. Choi Input feature selection for classification problems, IEEE Transactions on neural networks (2002) 13 (1).