Unsupervised and Transfer Learning Challenge

1. IJCNN 2011 San Jose, California Jul. 31, Aug. 5, 2011 Unsupervised and Transfer Learning Challenge Isabelle Guyon Clopinet, California http://clopinet.com/ul

2. Credits Data donors: Handwriting recognition (AVICENNA) -- Reza Farrahi Moghaddam, Mathias Adankon, Kostyantyn Filonenko, Robert Wisnovsky, and Mohamed Chériet (Ecole de technologie supérieure de Montréal, Quebec) contributed the dataset of Arabic manuscripts. The toy example (ULE) is the MNIST handwritten digit database made available by Yann LeCun and Corinna Costes. Object recognition (RITA) -- Antonio Torralba, Rob Fergus, and William T. Freeman, collected and made available publicly the 80 million tiny image dataset. Vinod Nair and Geoffrey Hinton collected and made available publicly the CIFAR datasets. See the techreport Learning Multiple Layers of Features from Tiny Images, by Alex Krizhevsky, 2009, for details. Human action recognition (HARRY) -- Ivan Laptev and Barbara Caputo collected and made publicly available the KTH human action recognition datasets. Marcin Marszałek, Ivan Laptev and Cordelia Schmid collected and made publicly available the Hollywood 2 dataset of human actions and scenes. Text processing (TERRY) -- David Lewis formatted and made publicly available the RCV1-v2 Text Categorization Test Collection. Ecology (SYLVESTER) -- Jock A. Blackard, Denis J. Dean, and Charles W. Anderson of the US Forest Service, USA, collected and made available the (Forest cover type) dataset. Challenge protocol and implemetation: Web platform: Server made available by Prof. Joachim Buhmann, ETH Zurich, Switzerland. Computer admin.: Peter Schueffler. Webmaster: Olivier Guyon, MisterP.net, France. Protocol review and advising: • David W. Aha, Naval Research Laboratory, USA. • Gideon Dror, Academic College of Tel-Aviv Yaffo, Israel. • Vincent Lemaire, Orange Research Labs, France. • Gavin Cawley, University of east Anglia, UK. • Olivier Chapelle, Yahoo!, California, USA. • Gerard Rinkus, Brandeis University, USA. • Ulrike von Luxburg, MPI, Germany. • David Grangier, NEC Labs, USA. • Andrew Ng, Stanford Univ., Palo Alto, California, USA • Graham Taylor, NYU, New-York. USA. • Quoc V. Le, Stanford University, USA. • Yann LeCun, NYU. New-York, USA. • Danny Silver, Acadia Univ., Canada. Beta testing and baseline methods: • Gideon Dror, Academic College of Tel-Aviv Yaffo, Israel. • Vincent Lemaire, Orange Research Labs, France. • Gregoire Montavon, TU Berlin, Germany. http://clopinet.com/ul

3. What is the problem? http://clopinet.com/ul

4. Can learning about... http://clopinet.com/ul

5. help us learn about… http://clopinet.com/ul

6. What is Transfer Learning? http://clopinet.com/ul

7. Vocabulary Target task labels Source task labels Target domain Source domain http://clopinet.com/ul

8. Target task labels Source task labels Target domain Source domain Vocabulary http://clopinet.com/ul

9. Target task labels Source task labels Target domain Source domain Vocabulary Labels available? Tasks the same? Domains the same? http://clopinet.com/ul

10. No labels in source domain Self-taught TL Inductive TL Labels available in source domain Multi-task TL Labels available in target domain Same source and target task Transductive TL Transfer Learning Labels avail. ONLY in source domain Semi-supervised TL Different source and target tasks Cross-task TL No labels in both source and target domains Unsupervised TL Taxonomy of transfer learning Adapted from: A survey on transfer learning, Pan-Yang, 2010. http://clopinet.com/ul

11. Challenge setting http://clopinet.com/ul

12. Challenge setting No labels in source domain Self-taught TL Inductive TL Labels available in source domain Multi-task TL Labels available in target domain Same source and target task Transductive TL Transfer Learning Labels avail. ONLY in source domain Semi-supervised TL Different source and target tasks Cross-task TL No labels in both source and target domains Adapted from: A survey on transfer learning, Pan-Yang, 2010. Unsupervised TL http://clopinet.com/ul

13. UTL Challenge Dec 2010-April 2011 http://clopinet.com/ul • Goal: Learning data representations or kernels. • Phase 1:Unsupervised learning (Dec 25, 2010-Mar 3, 2011) • Phase 2:Cross-task transfer learning (Mar. 4, 2011-Apr. 15, 2011) • Prizes: $6000 + free registrations + travel awards • Dissemination: ICML and IJCNN. Proceedings in JMLR W&CP. Development data Competitors Validation data Validation target task labels Data represen-tations Challenge data Evaluators Challenge target task labels http://clopinet.com/ul

14. UTL Challenge Dec 2010-April 2011 http://clopinet.com/ul • Goal: Learning data representations or kernels. • Phase 1:Unsupervised learning (Dec 25, 2010-Mar 3, 2011) • Phase 2:Cross-task transfer learning (Mar. 4, 2011-Apr. 15, 2011) • Prizes: $6000 + free registrations + travel awards • Dissemination: ICML and IJCNN. Proceedings in JMLR W&CP. Development data Competitors Source task labels Validation data Validation target task labels Data represen-tations Challenge data Evaluators Challenge target task labels http://clopinet.com/ul

15. Datasets of the challenge http://clopinet.com/ul

16. Evaluation http://clopinet.com/ul

17. AUC score For each set of samples queried, we assess the predictions of the learning machine with the Area under the ROC curve. http://clopinet.com/ul

18. Area under the Learning Curve(ALC) Linear interpolation. Horizontal extrapolation. http://clopinet.com/ul

19. Classifier used • Linear discriminant: f(x) = w . x = Si wi xi • Hebbian learning: X = (p, N) training data matrix Y {–1/p– , +1/p+}p target vector w = X’ Y = (1/p+)Skposxk –(1/p–) Sknegxk http://clopinet.com/ul

20. Kernel version • Kernel classifier: f(x) = Skak k(xk ,x) with a linear kernel k(xk ,x) = xk . x and with ak = –1/p– , if xk  neg ak = +1/p+ , if xk  pos • Equivalent linear discriminant f(x) = (1/p+)Skposxk . x – (1/p–) Sknegxk . x = w . x with w = (1/p+)Skposxk – (1/p–) Sknegxk http://clopinet.com/ul

21. Methods used http://clopinet.com/ul

22. No learning 1) Challenge platform Task labels P C prediction Pre- processed data Validation data http://clopinet.com/ul

23. No learning 1) Task labels P C prediction Pre- processed data Validation data Select the best preprocessing based on performance on the validation tasks http://clopinet.com/ul

24. No learning 1) P http://clopinet.com/ul

25. No learning Use the same preprocessor for the final evaluation 2) Task labels P C prediction Pre- processed data Challenge data http://clopinet.com/ul

26. R Unsupervised transfer learning 1) P Source domain Simultaneously train a preprocessor P and a re-constructor R using unlabeled data http://clopinet.com/ul

27. Unsupervised transfer learning 1) P http://clopinet.com/ul

28. Unsupervised transfer learning Use the same preprocessor for the evaluation on target domains 2) Task labels P C John Target domain http://clopinet.com/ul

29. Supervised data representation learning 1) Source task labels P C Sea Source domain Simultaneously train a preprocessor P and a classifier C with labeled source domain data http://clopinet.com/ul

30. Superviseddata representation learning Superviseddata representation learning 1) P http://clopinet.com/ul

31. Superviseddata representation learning Use the same preprocessor for the evaluation on target domains 2) Task labels P C John Target domain http://clopinet.com/ul

32. Variants • Use all or subsets of data for training (development/validation/challenge data). • Learn what preprocessing steps to apply w. validation data (not the preprocessor) then apply the method to challenge data. • Learn to reconstruct noisy versions of the data. • Train a kernel instead of a preprocessor. http://clopinet.com/ul

33. Results http://clopinet.com/ul

34. Questions • Can Transfer Learning beat raw data (or simple preprocessing)? • Does Deep Learning work? • Do labels help (does cross-task TL beat unsupervised TL)? • Is model selection possible in TL? • Did consistent TL methodologies emerge? • Do the results make sense? • Is there code available? http://clopinet.com/ul

35. Can transfer learning beat raw data? Phase 2 (1141 jobs sumitted, 14 complete final entries) Phase 1 (6933 jobs submitted, 41 complete final entries) http://clopinet.com/ul

36. Results (ALC) http://clopinet.com/ul

37. Does “Deep Learning” work? Evolution of performance as a function of depth on SYLVESTER LISA team, 1st in phase 2, 4th in phase 1 http://clopinet.com/ul

38. Do labels help in TL? http://clopinet.com/ul

39. Is model selection possible? Phase 2 Phase 1 Use of “transfer labels”: the -criterion (LISA team) http://clopinet.com/ul

40. Did consistent methodologies emerge? http://clopinet.com/ul

41. Results (ALC) http://clopinet.com/ul

42. Bottom layers:Preprocessing and feature selection http://clopinet.com/ul

43. Middle layers http://clopinet.com/ul

44. Top layer http://clopinet.com/ul

45. Implementation http://clopinet.com/ul

46. A few things that worked well • Learn the preprocessing steps (not the preprocessor) – Aiolli, 1st phase 1. • As 1st steps: eliminate low info features or keep largest PC and sphere the data, normalize, and/or standardize. • Learn denoising or contrastive auto-encoders or RBMs– LISA team, 1st phase 2. • Use cluster memberships of multiple K-means – 1055A team, 2nd phase 1 and 3rd phase 2. • Transductive PCA (as last step) – LISA. http://clopinet.com/ul

47. Conclusion • UL:This challenge demonstrated the potential of unsupervised learning methods used as preprocessing to supervised learning tasks. • UTL:Model selection of UL hyper-parameters can be carried out with “source tasks” similar to the “target tasks”. • DL: Multi-step preprocessing leading to deep architectures can be trained in a greedy bottom-up step-wise manner. • Favorite methods include normalizations, PCA, clustering, and auto-encoders. • A kernel method won phase 1 and a Deep Learning method won phase 2. http://clopinet.com/ul

48. Gesture Recognition Challenge June 2011-June 2012 http://gesture.chalearn.org STEP 1: Develop a “generic” gesture recognition system that can learn new signs with a few examples. STEP 2: At conference: teach the system new signs. STEP 3: Live evaluation in front of audience. http://clopinet.com/ul