180 likes | 312 Views
Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng ICML 2009. Presented by: Mingyuan Zhou Duke University, ECE September 18, 2009. Outline. Motivations Contributions
E N D
Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng ICML 2009 Presented by: Mingyuan Zhou Duke University, ECE September 18, 2009
Outline • Motivations • Contributions • Backgrounds • Algorithms • Experiment results • Deep Vs Shallow • Conclusions
Motivations • To Learn hierarchical models which simultaneously represent multiple levels, e.g., pixel intensities, edges, object parts, objects, and beyond can be represented by layers from low to high. • Combining top-down and bottom-up processing of an image. • Limitations of deep belief networks (DBNs) • Scaling DBNs to realistic-size images remains challenging: images are high-dimentional and objects can appear at arbitrary locations in images.
Contributions • Convolutional RBM: feature detectors are shared among all locations in an image. • Probabilistic max-pooling: in a probabilistic sound way allowing higher-layer units to cover larger areas of the input. • The first translation invariant hierarchical generative model supporting both top-down and bottom-up probabilistic inference and sales to realistic image sizes.
Backgrounds: Restricted Boltzmann Machine (RBM) • Giving the visible layer, the hidden units are conditionally independent, and vise versa. • Efficient block Gibbs sampling can be performed by alternately sampling each layer’s units. • Computing the exact gradient of the log-likelihood is intractable, so the contrastive divergence approximation is commonly used. (binary v) (real-value v)
Backgrounds: Deep belief network (DBN) • In a DBN, two adjacent layers have a full set of connections between them, but no two units in the same layer are connected. • A DBN can be formed by stacking RBMs. • An efficient algorithm for training DBNs (Hinton et al., 2006): greedily training each layer, from lowest to highest, as an RBM using the previous layer's activations as inputs.
Algorithms: Probabilistic max-pooling • Each unit in a pooling layercomputes the maximum activation of the units in asmall region of the detection layer. • Shrinking the representationwith max-pooling allows higher-layer representationsto be invariant to small translations of theinput and reduces the computational burden. • Max-pooling was intended only for feed-forward architectures. A generative model of images which supports both top-down and bottom-up inference is of interest.
Algorithms: Sparsity regulations • Only a tiny fractionof the units should be active in relation to a givenstimulus. • Regularizing the objective function toencourageeach of the hidden units to have a mean activationclose to some small constant .
Algorithms: Convolutional DBN (CDBN) • CDBN consists of several max-pooling-CRBMs stacked on top of one another. • Once a given layer istrained, its weights are frozen, and its activations areused as input to the next layer.
Deep Vs Shallow From Jason Weston’s slides: DEEPLEARNING VIASEMI-SUPERVISED EMBEDDING, ICML 2009 WORKSHOP ON LEARNING FEATURE HIERARCHIES . From Francis Bach’s slides:Convex sparse methods for feature hierarchies, ICML 2009WORKSHOP ON LEARNING FEATURE HIERARCHIES
Conclusions Convolutional deep belief network: • A scalable generative model for learning hierarchical representations from unlabeled images. • Performing well in a variety of visual recognition tasks.