Incremental Learning with Multiple Classifier Systems using Correction Filters for Classification

1. Incremental Learning withMultiple Classifier Systems usingCorrection Filters for Classification Authors: José del Campo Ávila, Gonzalo Ramos Jiménez, Rafael Morales Bueno IDA 2007, Ljubljana (Slovenia)

2. Outline • Introduction • Incremental learning with Multiple Classifier Systems • MultiCIDIM-DS • Correction Filters for Classification • MultiCIDIM-DS-CFC • Experiments and results • Conclusion Introduction Incremental MCS Correction Filters Experiments Conclusion

3. Introduction • Multiple classifier systems (MCS) • Ensemble of basic models • Main requirements of the basic models • Accuracy • Diversity • Advantages • Increased expressive ability • Higher accuracy • Very large datasets or data streams • Incremental learning • Concept drift • Use of Correction Filters for Classification (CFC) • Independent of MCS approach • Can be applied to a wide variety of MCS Introduction Incremental MCS Correction Filters Experiments Conclusion

4. Incremental learning with MCS • Generation of MCS from large datasets • Sequential • Parallel • Advantages • Lower memory requirements • Increases diversity • Concept drift Introduction Incremental MCS Correction Filters Experiments Conclusion

5. Incremental learning with MCS • Combining basic models of MCS • Combination • Voting • Fusion • Selection of basic models • Static • Dynamic • Advantages • Any-time learning Introduction Incremental MCS Correction Filters Experiments Conclusion

6. Incremental learning with MCS • MultiCIDIM-DS • Base algorithm: CIDIM • Generation of base classifiers: sequential • Combination of base classifiers: fusion • Selection of base classifiers: static • Advantages • High accuracy of base classifiers • Simplicity of base classifiers Introduction Incremental MCS Correction Filters Experiments Conclusion

7. Incremental MCS with Correction Filters • Correction Filters for Classification • Identify which subspaces are correctly learnt by base classifiers • New classifiers are trained (CFC) • New more informed combining method can be used Introduction Incremental MCS Correction Filters Experiments Conclusion

8. Incremental MCS with Correction Filters • CFC are induced by incremental algorithms • None information is discarded • Detection of concept drift (if available) • Using of parallel approach Introduction Incremental MCS Correction Filters Experiments Conclusion

9. Incremental MCS with Correction Filters • Dynamic combination of base classifier predictions • Includes new information stored in CFC • Only information of relevant base classifiers is used • Some base classifier is relevant  only best classifiers are used • None base classifier is relevant  use of minority class Introduction Incremental MCS Correction Filters Experiments Conclusion

10. Incremental MCS with Correction Filters • MultiCIDIM-DS-CFC • Base algorithm: CIDIM • Incremental algorithm: IADEM-2 • Generation of base classifiers: sequential • Generation of incremental classifiers: parallel • Combination of base classifiers: fusion • Selection of base classifiers: dynamic • Advantages • High accuracy of base classifiers • Simplicity of base classifiers • More informed predictions • Preserves any-time learning Introduction Incremental MCS Correction Filters Experiments Conclusion

11. Experiments and results • Algorithms used in the experiments • VFDT • Bagging using C4.5 (J48 from Weka) • Boosting using C4.5 (J48 from Weka) • MultiCIDIM-DS • MultiCIDIM-DS-CFC • Datasets • Synthetic • Syn_1: 0% noise and balanced • Syn_2: 0% noise and imbalanced • Syn_3: 15% noise and balanced • Real • Electricity • LED Display Introduction Incremental MCS Correction Filters Experiments Conclusion

12. Less improvement Greater improvement Experiments and results • Behaviour depending on block size Introduction Incremental MCS Correction Filters Experiments Conclusion

13. Experiments and results Introduction Incremental MCS Correction Filters Experiments Conclusion

14. Conclusions • Multiple classifier systems for incremental learning • Extend the benefits of MCS to the incremental learning approach (higher accuracy) • Solve the limitation of high memory requirements of traditional classifier algorithms • Correction Filters for Classification (CFC) • Detect which subspaces are correctly classified by the base classifiers • Allow to do a more informed combination of the base classifiers to get a prediction Introduction Incremental MCS Correction Filters Experiments Conclusion

15. Conclusions • Advantages • Improvement of the accuracy using CFC • Preservation of incremental learning • Flexibility (applicable to other algorithms) • Limitations • Computational cost Introduction Incremental MCS Correction Filters Experiments Conclusion

16. Future Work • Study the performance of CFC with other multiple classifier systems • Study more extensively the behaviour of CFC • Block size • Algorithms • Basic • Incremental • Include concept drift detection Introduction Incremental MCS Correction Filters Experiments Conclusion

17. Incremental Learning withMultiple Classifier Systems usingCorrection Filters for Classification Authors: José del Campo Ávila, Gonzalo Ramos Jiménez, Rafael Morales Bueno IDA 2007, Ljubljana