Popular Ensemble Methods: An Empirical Study

1. Popular Ensemble Methods: An Empirical Study David Opitz and Richard Maclin Presented by Scott Wespi 5/22/07

2. Outline • Ensemble methods • Classifier Ensembles • Bagging vs Boosting • Results • Conclusion

3. Ensemble Methods • Sets of individually trained classifiers whose predictions are combined when classifying new data • Bagging (1996) • Boosting (1996) • How are bagging and boosting influenced by the learning algorithm? • Decision trees • Neural networks *Note: Paper is from 1999

4. Classifier Ensembles • Goal: highly accurate individual classifiers that disagree as much as possible • Bagging and boosting create disagreement

5. Bagging vs. Boosting

6. Ada-Boosting vs Arcing • Ada-Boosting • Every sample has 1/N weight initially, increases every time sample was skipped or misclassified • Arcing • If mi = number of times ith example was misclassified

8. Neural Networks • Ada-Boosting • Arcing • Bagging • White bar represents 1 • standard deviation

9. Decision Trees

10. Composite Error Rates

11. Neural Networks: Bagging vs Simple

12. Ada-Boost: Neural Networks vs. Decision Trees • NN • DT • Box represents • reduction in error

13. Arcing

14. Bagging

15. Noise • Hurts boosting the most

16. Conclusions • Performance depends on data and classifier • In some cases, ensembles can overcome bias of component learning algorithm • Bagging is more consistent than boosting • Boosting can give much better results on some data