Presented by: Mingyuan Zhou Duke University, ECE June 17, 2011

1. An Analysis of Single-Layer Networksin Unsupervised Feature LearningAdam Coates, Honglak Lee and Andrew Y. NgAISTATS 2011The Importance of Encoding Versus Training with Sparse Coding and Vector QuantizationAdam Coates and Andrew Y. NgICML 2011 Presented by: Mingyuan Zhou Duke University, ECE June 17, 2011

2. Outline An Analysis of Single-Layer Networksin Unsupervised Feature LearningAdam Coates, Honglak Lee and Andrew Y. NgAISTATS 2011 • Introduction • Unsupervised feature learning • Parameter setting • Experiments on CIFAR, NORB and STL • Conclusions

3. Training/testing pipeline • Feature learning: • Extract random patches from unlabeled training images • Apply a pre-processing stage to the patches • Learn a feature-mapping using an unsupervised learning algorithm • Feature extraction and classification: • Extract features from equally spaced sub-patches covering the input image • Pool features together over regions of the input image to reduce the number of feature values • Train a linear classifier to predict the labels given the feature vectors

4. Feature learning • Pre-processing of patches • Mean subtraction and scale normalization • Whitening • Unsupervised learning • Sparse auto-encoder • Sparse restricted Boltzmann machine • K-means clustering Hard-assignment: Soft-assignment:

5. Feature learning • Unsupervised learning • Sparse auto-encoder • Sparse restricted Boltzmann machine • K-means clustering • Gaussian mixture model (GMM)

6. Feature extraction and classification

7. Experiments and analysis • Model parameters: • Whitening? • Number of features K • Stride s (all the overlapping patches are used when s = 1) • Receptive field (patch) size w

8. Experiments and analysis

9. Experiments and analysis

10. Experiments and analysis

11. Experiments and analysis

12. Conclusions Mean-subtraction, scale normalization and Whitening + Large K + Small s + Right patch size w + Simple feature learning algorithm (soft K-means) = State-of-the-art results on CIFAR-10 and NORB

13. Outline The Importance of Encoding Versus Training with Sparse Coding and Vector QuantizationAdam Coates and Andrew Y. NgICML 2011 • Motivations and contributions • Review of dictionary learning algorithms • Review of sparse coding algorithms • Experiments on CIFAR, NORB and Caltech101 • Conclusions

14. Main contributions

15. Dictionary learning algorithms • Sparse coding (SC) • Orthogonal matching pursuit (OMP-k) • Sparse RBMs and sparse auto-encoders (RBM, SAE) • Randomly sampled patches (RP) • Random weights (R)

16. Sparse coding algorithms • Sparse coding (SC) • OMP-k • Soft threshold (T) • “Natural” encoding

17. Experimental results

18. Experimental results

22. Comments on dictionary learning • The results have shown that the main advantage of sparse coding is as an encoder, and that the choice of basis functions has little effect on performance. • The main value of the dictionary is to provide a highly overcomplete basis on which to project the data before applying an encoder, but that the exact structure of these basis functions is less critical than the choice of encoding • All that appears necessary is to choose the basis to roughly tile the space of the input data. This increases the chances that a few basis vectors will be near to an input, yielding a large activation that is useful for identifying the location of the input on the data manifold later • This explains why vector quantization is quite capable of competing with more complex algorithms: it simply ensures that there is at least one dictionary entry near any densely populated areas of the input space. We expect that learning is more crucial if we use small dictionaries, since we would then need to be more careful to pick basis functions that span the space of inputs equitably.

23. Conclusions • The main power of sparse coding is not that it learns better basis functions. In fact, we discovered that any reasonable tiling of the input space (including randomly chosen input patches) is sufficient to obtain high performance on any of the three very different recognition problems that we tested. • Instead, the main strength of sparse coding appears to arise from its non-linear encoding scheme, which was almost universally effective in our experiments—even with no training at all. • Indeed, it was difficult to beat this encoding on the Caltech 101 dataset. In many cases, however, it was possible to do nearly as well using only a soft threshold function, provided we have sufficient labeled data. • Overall, we conclude that most of the performance obtained in our results is a function of the choice of architecture and encoding, suggesting that these are key areas for further study and improvements.