1 / 35

Classifier ensembles: Does the combination rule matter?

Classifier ensembles: Does the combination rule matter?. Ludmila Kuncheva School of Computer Science Bangor University, UK l.i.kuncheva@bangor.ac.uk. class label. classifier ensemble. combiner. classifier. classifier. classifier. feature values (object description).

adele-harding
Download Presentation

Classifier ensembles: Does the combination rule matter?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Classifier ensembles: Does the combination rule matter? LudmilaKuncheva School of Computer Science Bangor University, UK l.i.kuncheva@bangor.ac.uk

  2. class label classifier ensemble combiner classifier classifier classifier feature values (object description)

  3. Congratulations! The Netflix Prize sought to substantially improve the accuracy of predictions about how much someone is going to enjoy a movie based on their movie preferences. On September 21, 2009 we awarded the $1M Grand Prize to team “BellKor’s Pragmatic Chaos”. Read about their algorithm, checkout team scores on the Leaderboard, and join the discussions on the Forum. We applaud all the contributors to this quest, which improves our ability to connect people to the movies they love. class label classifier ensemble combiner classifier classifier classifier feature values (object description)

  4. class label classifier ensemble combiner classifier classifier classifier cited 7194 times by 28 July 2013 (Google Scholar) feature values (object description)

  5. Classifier combination? Hmmmm….. David J. Hand (2006) Classifier technology and the illusion of progress, Statist. Sci. 21(1), 1-14. SasoDzeroski We are kidding ourselves; there is no real progress in spite of ensemble methods. S. Dzeroski, and B. Zenko. (2004) Is combining classifiers better than selecting the best one? Machine Learning, 54, 255-273. Chances are that the single best classifier will be better than the ensemble. David Hand

  6. Quo Vadis? "combining classifiers" OR "classifier combination" OR "classifier ensembles" OR "ensemble of classifiers" OR "combining multiple classifiers" OR "committee of classifiers" OR "classifier committee" OR "committees of neural networks" OR "consensus aggregation" OR "mixture of experts" OR "bagging predictors" OR adaboost OR (( "random subspace" OR "random forest" OR "rotation forest" OR boosting) AND "machine learning")

  7. Gartner’s Hype Cycle: a typical evolution pattern of a new technology Where are we?...

  8. (6) IEEE TPAMI = IEEE Transactions on Pattern Analysis and Machine Intelligence IEEE TSMC = IEEE Transactions on Systems, Man and Cybernetics JASA = Journal of the American Statistical Association IJCV = International Journal of Computer Vision JTB = Journal of Theoretical Biology (2) PPL = Protein and Peptide Letters JAE = Journal of Animal Ecology PR = Pattern Recognition (4) ML = Machine Learning NN = Neural Networks CC = Cerebral Cortex top cited paper is from… application paper

  9. International Workshop on Multiple Classifier Systems 2000 – 2013 - continuing

  10. Levels of questions • A Combination level • selection or fusion? • voting or another combination method? • trainable or non-trainable combiner? Combiner • B Classifier level • same or different classifiers? • decision trees, neural networks or other? • how many? Classifier 1 Classifier 2 … Classifier L • CFeature level • all features or subsets of features? • random or selected subsets? Features • DData level • independent/dependent bootstrap samples? • selected data sets? Data set

  11. Strength of classifiers The perfect classifier ?  Large ensemble of nearly identical classifiers - REDUNDANCY • 3-8 classifiers • heterogeneous • trained combiner • (stacked generalisation) • 30-50 classifiers • same or different models? • trained or non-trained combiner? • selection or fusion? • IS IT WORTH IT? How about here? Number of classifiers L 1 Must engineer diversity…  Small ensembles of weak classifiers - INSUFFICIENCY ? • 100+ classifiers • same model • non-trained combiner • (bagging, boosting, etc.)

  12. Strength of classifiers The perfect classifier  Large ensemble of nearly identical classifiers - REDUNDANCY • 3-8 classifiers • heterogeneous • trained combiner • (stacked generalisation) Diversity is pretty impossible… • 30-50 classifiers • same or different models? • trained or non-trained combiner? • selection or fusion? • IS IT WORTH IT? Number of classifiers L 1 Must engineer diversity…  Small ensembles of weak classifiers - INSUFFICIENCY • 100+ classifiers • same model • non-trained combiner • (bagging, boosting, etc.) Diversity is absolutely CRUCIAL!

  13. Label outputs Continuous-valued outputs Decision profile 1 2 3 1 2 3 x x

  14. Ensemble (label outputs, R,G,B) Red Red Blue Green Majority vote Red Red Red 204 R 102 G 54 B

  15. Majority vote Ensemble (label outputs, R,G,B) Red Red Red Blue Green Red Red 200 R 219 G 190 B Weighted Majority vote Green 0.27 0.70 0.50 0.05 0.50 0.02 0.10 0.70 0.10

  16. RBRRGR Ensemble (label outputs, R,G,B) Red Red Blue Classifier Green Red Red Green

  17. Ensemble (continuous outputs, [R,G,B]) [0.7 0.6 0.5] [0.1 0.0 0.6] [0.4 0.3 0.1] [0.6 0.3 0.1] [0 1 0] [0.9 0.7 0.8]

  18. Ensemble (continuous outputs, [R,G,B]) [0.7 0.6 0.5] [0.1 0.0 0.6] [0.4 0.3 0.1] [0.6 0.3 0.1] [0 1 0] [0.9 0.7 0.8] Mean R = 0.45

  19. Ensemble (continuous outputs, [R,G,B]) [0.70.6 0.5] [0.10.0 0.6] [0.4 0.3 0.1] [0.60.3 0.1] [0 1 0] [0.90.7 0.8] Mean R = 0.45 Mean G = 0.48

  20. Ensemble (continuous outputs, [R,G,B]) [0.70.60.5] [0.10.00.6] [0.4 0.3 0.1] [0.60.30.1] [0 1 0] [0.90.7 0.8] Mean R = 0.45 Mean G = 0.48 Mean B = 0.35 Class GREEN

  21. Ensemble (continuous outputs, [R,G,B]) [0.70.60.5] [0.10.00.6] Decision profile [0.4 0.3 0.1] 0.6 0.3 0.1 0.1 0.0 0.6 0.7 0.6 0.5 0.4 0.3 0.1 0 .0 1.0 0.0 0.9 0.7 0.8 [0.60.30.1] [0 1 0] [0.90.7 0.8] Mean R = 0.45 Mean G = 0.48 Mean B = 0.35 Class GREEN

  22. Decision profile classes Support that classifier #4 gives to the hypothesis that the object to classify comes from class #3. 0.6 0.3 0.1 0.1 0.0 0.6 0.7 0.6 0.5 0.4 0.3 0.1 0 .0 1.0 0.0 0.9 0.7 0.8 classifiers Would be nice if these were probability distributions...

  23. Decision profile classes classifiers … We can take probability outputs from the classifiers

  24. Combination Rules For label outputs For continuous-valued outputs Majority (plurality) vote Simple rules: minimum, maximum, product, average (sum) Weighted majority vote Naïve Bayes c Regressions BKS A classifier A classifier

  25. Combination Rules Decision profile For label outputs For continuous-valued outputs Majority (plurality) vote Simple rules: minimum, maximum, product, average (sum) Weighted majority vote Naïve Bayes c Regressions BKS A classifier A classifier

  26. class label classifier ensemble combiner classifier classifier classifier feature values (object description)

  27. class label classifier ensemble classifier classifier classifier classifier feature values (object description)

  28. Bob Duin: The Combining Classifier: to Train or Not to Train? http://samcnitt.tumblr.com/

  29. Tin Ho: “Multiple Classifier Combination: Lessons and Next Steps”, 2002 “Instead of looking for the best set of features and the best classifier, now we look for the best set of classifiers and then the best combination method. One can imagine that very soon we will be looking for the best set of combination methods and then the best way to use them all. If we do not take the chance to review the fundamental problems arising from this challenge, we are bound to be driven into such an infinite recurrence, dragging along more and more complicated combination schemes and theories and gradually losing sight of the original problem.”

  30. Conclusions - 1 • Classifier ensembles: Does the combination rule matter? • In a word, yes. • But its merit depends upon • the base classifier model, • the training of the individual classifiers, • the diversity, • the possibility to train the combiner, and more.

  31. Conclusions - 2 The choice of the combiner should not be side-lined. The combiner should be chosen in relation to the rest of the ensemble and the available data.

  32. Questions to you: What is the future of classifier ensembles? (Are they here to stay or are they a mere phase?) In what direction(s) will they evolve/dissolve? What will be the ‘classifier of the future’? Or the ‘classification paradigm of the future’? And one last question: How can we get a handle of the ever growing scientific literature in each and every area? How can we find the gems among the pile of stones?

More Related