Student : Chuan LU Promoters: Prof. Dr. Ir. Sabine Van Huffel

1. Predictive Computer Models for Medical Classification ProblemsPhD progress report(2000.8 ~ 2003.10) Student : Chuan LU Promoters: Prof. Dr. Ir. Sabine Van Huffel Prof. Dr. Ir. Johan Suykens Advisers : Prof. Dr. Dirk Timmerman Prof. Dr. Ir. Joos Vandewalle Prof. Dr. Jan Beirlant PhD Hearing (Oct 15, 2003)

2. Overview • PhD topic • Doctoral Programme: • courses, publication, meetings... • Research Work • Work plan and timing PhD Hearing (Oct 15, 2003)

3. PhD Topic • The development, statistical analysis and clinical evaluation of a new class of predictive models which optimally extract information from patient data. • The attention is focused on intelligence machine learning methods such as neural networks, kernel based algorithms, and their integration with Bayesian framework. PhD Hearing (Oct 15, 2003)

4. Joint Research Activities • Classification of ovarian tumors • logistic regression (LR) • artificial neural networks (ANNs) • Bayesian least squares support vector machines (LS-SVMs) • Prediction of pregnancy of unknown location (PUL) • LR, LS-SVMs, relevance vector machines (RVMs) • Variable selection for medical classification problems: (Bayesian framework) PhD Hearing (Oct 15, 2003)

5. The Doctoral Programme • ‘Direct Tuition’ – (520 h) • Doctoral Courses • Case Studies in Biomedical Data Processing (25x6=150 h) • Phd training course: Longitudinal Data,Incomplete Data, and Causal Inference (18x1=18 h) • Courses in master of statistics • Basic Concepts of Statistical Modeling (60x4=240 h) • Applied Statistical Models (60x4=240 h) • Seminars • Presentation at BioMed Seminar, SISTA (50 x 1 h) • Presentation at SISTA seminar on Feb 28, 2002 (50x0.5 h) PhD Hearing (Oct 15, 2003)

6. The Doctoral Programme • Other Study Activities and Achievements • Publications (1) – with first authorship [1]‘Preoperative prediction of malignancy of ovarian tumors using least squares support vector machines’. Artificial Intelligence in Medicine, vol. 28, no. 3, Jul. 2003, pp. 281-306. (200 h) [2] ‘Using Artificial Neural Networks to Predict Malignancy of Ovarian Cancers’, in Proc. Of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society - EMBC2001, Istanbul, Turkey, Oct. 2001, CD-ROM. (100 h) [3] ‘Classification of ovarian tumor using Bayesian least squares support vector machines’, accepted for publication in the 9th Conference on Artificial Intelligence in Medicine Europe (AIME 03), Oct 18-22, Cyprus. (100+60 h) [4] ‘Bayesian Least Squares Support Vector Machines for Classification of Ovarian Tumors’, Internal Report 02-105, ESAT-SISTA, K.U.Leuven (Leuven, Belgium), 2002. PhD Hearing (Oct 15, 2003)

7. The Doctoral Programme • Publications (2) – with coauthorship [1] ‘Prediction of mental development of preterm newborns at birth time using LS-SVM’, in Proc. of ESANN'02, Bruges, Belgium, Apr. 2002, pp. 167-172. [2] ‘Color Doppler and Gray-Scale Ultrasound Evaluation of the Postpartum uterus’, Ultrasound Obstet. Gynecol., vol. 20, 2002, pp. 586-591 [3] ‘Prospective evaluation of blood flow in the myometrium and in the uterine arteries in the puerperium’, Jan 2003, accepted for publication in Ultrasound in Obstetrics and Gynecology. [4] ‘Subjective use of serum human chorionic gonadotrophin and progesterone levels for the investigation of pregnancies of unknown location : analysis of interuser variability and experience’, 2003. [5] ‘Can ectopic pregnancies be predicted using serum hormone levels ?’, submitted, 2003. [6] ‘A novel neural approach to inverse problems with discontinuities (the GMR neural network)’, in IJCNN'03, Portland, Oregon, Jul. 2003, pp. 3106-3111. [7] ‘Direct Torque control of induction motors by use of the GMR Neural network’, in IJCNN'03, Portland, Oregon, July 20-24, 2003, pp. 2106-2111. PhD Hearing (Oct 15, 2003)

8. The Doctoral Programme • Other Study Activities and Achievements • Participation in Scientific Meetings • 9th Annual Meeting of the Belgian Statistical Society-BSS2001, Oostende, Oct 2001, Poster Presentation: ‘Prediction of Malignancy of Ovarian Tumors Using Logistic Regression and Artificial Neural Network Models’ • the Advanced NATO Study Institute on Learning Theory and Practice (NATO-ASI LTP 2002), Leuven, Belgium, July 8-19, 2002, Poster Presentation: ‘Blackbox classifiers for preoperative discrimination between malignant and benign ovarian tumors’. • 10th Annual Meeting of the Belgian Statistical Society-BSS2002, Kerkrade, The Netherlands, October 18-19, 2002Poster Presentation: ‘Comparative study on variable selection for nonlinear classifiers’ • Poster presentation at study day of IAP network 2001, 2002 and 2003. • Belgian Day of Biomedical Engineering in Brussels, , October 17th 2003, Poster presentation: ‘Variable selection using linear sparse Bayesian models for medical classification problems’. PhD Hearing (Oct 15, 2003)

9. The Doctoral Programme • Other Study Activities and Achievements • Supervision of licentiate thesis (2x150+100=400 h) • Thesis supervision for Master of applied statistics: ‘Mathematical models for predicting the evolution of a pregnancy of unknown location’, 2003 • Thesis supervision for Master of applied statistics: ‘Prediction of pregnancy evolution and ectopic pregnancy’ , 2002. • Thesis supervision for ERASMUS student from university de Picardie Jules Verne, France, 2003, Approches neuronales pour la résolution de problèmes inverses avec discontinuités PhD Hearing (Oct 15, 2003)

10. Research - Building blocks • Explorative data analysis (EDA) • Probabilistic modeling techniques • Variable selection • Applications PhD Hearing (Oct 15, 2003)

11. Explorative Data Analysis • Gain insights into a data set: structure, imprtant variables, outiers, and the model suggested by the data. • Techniques: scatterplots, boxplots, histograms, PCA, FA, CCA, biplots, etc. • New nonlinear techniques in EDA: kernel PCA, kernel CCA, and nonlinear biplots. Uncovering the nonlinear structure of the data, aid in nonlinear modeling such as LS-SVM. PhD Hearing (Oct 15, 2003)

12. Fig. Biplot of Ovarian Tumor data. • The observations are plotted as points (o - benign, x - malignant), the variables are plotted as vectors from the origin. • - visualization of the correlation between the variables • - visualization of the relations between the variables and clusters. Explorative Data Analysis PhD Hearing (Oct 15, 2003)

13. Explorative Data Analysis • Fig. Nonlinear Biplot for kernel PCA with RBF kernels • Data projected onto pairs of PCs (PCs with the largest correlation with y were selected for visualization) , computed by kernel trick. • Approximate decision boundary: ridge regression (y=1) using pairs of PCs • For kth variable, pseudosamples generated by: setting data mean as starting point, varying the value of variable k while fixing the others. Variable trajectory: tracing the projection of the pseudosample onto the pairs of PCs. PhD Hearing (Oct 15, 2003)

14. Probabilistic Modeling • Probabilistic modeling needed in medical Decis. Supp.  the uncertainty and different mis. class. cost. • Traditional statistical linear probabilistic classifiers: • Linear discriminant analysis (LDA) • Logistic regression (LR) • Bayesian MLPs • Bayesian + Kernel based modeling: • Bayesian LS-SVM classifiers (Suykens 1999, 2001, 2002) • Sparse Bayesian modeling and relevance vector machines (RVMs) (Tipping 2001,2003) PhD Hearing (Oct 15, 2003)

15. Recipe • Goal: • Linear model y=wTx Nonlinear model • Dealing with uncertainty • Model selection • Sparseness • Ingredients: • Kernel trick: x(x) higher dim. feature space • Bayesian framework PhD Hearing (Oct 15, 2003)

16. Bayesian Inference • Find the maximum a posterior (MAP) estimates of model parameters wMP and bMP, using conventional LS-SVM training. • The posterior probability of the parameters can be estimated via marginalization using Gaussian probability at wMP, bMP • Assuming a uniform prior p(Hj) over all model, rank the model by the evidence p(D|Hj) evaluated using Gaussian approximation. PhD Hearing (Oct 15, 2003)

17. Variable selection • Importance in medical classification problems • economics of data acquisition • accuracy and complexity of the classifiers • gain insights into the underlying medical problem. • Filter approaches: filter out irrelevant attributes before induction occurs • Wrapper approaches: focus on finding attributes that are usefulfor performance for a specific type of model, rather than necessarily finding the relevant ones. PhD Hearing (Oct 15, 2003)

18. Variable selection • Heuristic search: • forward, backward,stepwise • hill-climbing, branch and bound… • Variable selection criteria: • Correlation, fisher score, • mutual information • Evidence in Bayesian framework • Classification performance, e.g. AUC • Sensitivity analysis: change in the objective function J by removing variable i: DJ(i) • Statistical chi-square test PhD Hearing (Oct 15, 2003)

19. Variable selection • We focus on evidence (marginal likelihood) based method within the Bayesian framework • Forward / stepwise selection • Bayesian LS-SVM • Sparse Bayesian models • Accounting for uncertainty in variable selection PhD Hearing (Oct 15, 2003)

20. Application- Ovarian tumor classification • Problem • develop a reliable diagnostic tool to discriminate preoperatively between benign and malignant tumors. • assist clinicians in choosing the appropriate treatment. • Data (from IOTA project) • Patient data collected at Univ. Hospitals Leuven, Belgium, 1994~1999 • 425 records, 25 features. • 291 benign tumors, 134 (32%) malignant tumors. PhD Hearing (Oct 15, 2003)

21. Evolution of the model evidence Application- Ovarian tumor classification • Forward variable selection based on Bayesian LS-SVM 10 variables were selected based on the training set (first treated 265 patient data) using RBF kernels. PhD Hearing (Oct 15, 2003)

22. Application- Ovarian tumor classification • Predictive power of the models given the selected variables ROC curves on test Set (data from 160 newest treated patients) PhD Hearing (Oct 15, 2003)

23. Application- Ovarian tumor classification • Performance on test set with rejection based on, e.g., • The rejected patients need further examination by human experts • Posterior probability essential for medical decision making PhD Hearing (Oct 15, 2003)

24. cancer no. samples no. genes task leukemia 72 7192 2 subtypes colon 62 2000 disease/normal Application- binary cancer classification based on microarray data • Variable selection using linear sparse Bayesian logit model, LOO CV accuracy PhD Hearing (Oct 15, 2003)

25. Application- brain tumor multiclass classification based on MRS spectra data • 4 types of brain tumors, 205x138 magnitude value • Variable selection using linear sparse Bayesian logit model, 30 runs of random CV accuracy PhD Hearing (Oct 15, 2003)

26. Work Plan and Timing • Nov - Overview paper on Linear and nonlinear preoperative classification of ovarian tumors (chapter proposal accepted for edited book "Knowledge Based Intelligent System for Health Care.") • Dec – Jan 2003, paper on variable selection • Feb 2004 – model averaging • Model evaluation using IOTA data. • April 2004 – writing draft of thesis • Sept 2004 - Defense PhD Hearing (Oct 15, 2003)