Deconvolutional Networks

1. Deconvolutional Networks • Matthew D. ZeilerDilip Krishnan, Graham W. TaylorRob Fergus • Dept. of Computer Science, Courant Institute, New York University

2. Matt Zeiler

3. Overview • Unsupervised learning ofmid and high-level image representations • Feature hierarchy built from alternating layers of: • Convolutional sparse coding (Deconvolution) • Max pooling • Application to object recognition

4. Motivation • Good representations are key to many tasks in vision • Edge-based representations are basis of many models • SIFT [Lowe’04], HOG [Dalal & Triggs ’05] & others Yan & Huang (Winner of PASCAL 2010 classification competition) Felzenszwalb, Girshick, McAllester and Ramanan, PAMI 2007

5. Beyond Edges? • Mid-level cues • Continuation • Parallelism • Junctions • Corners • “Tokens” from Vision by D.Marr • High-level object parts:

6. Two Challenges • Grouping mechanism • Want edge structures to group into more complex forms • But hard to define explicit rules • Invariance to local distortions • Corners, T-junctions, parallel lines etc. can look quite different

7. Talk Overview • Single layer • Convolutional Sparse Coding • Max Pooling • Multiple layers • Multi-layer inference • Filter learning • Comparison to related methods • Experiments

8. Talk Overview • Single layer • Convolutional Sparse Coding • Max Pooling • Multiple layers • Multi-layer inference • Filter learning • Comparison to related methods • Experiments

9. Recap: Sparse Coding (Patch-based) • Over-complete linear decompositionof input using dictionary Input Dictionary • regularization yields solutionswith few non-zero elements • Output is sparse vector:

10. Single Deconvolutional Layer • Convolutional form of sparse coding

11. Single Deconvolutional Layer

12. Single Deconvolutional Layer

13. Single Deconvolutional Layer 1

14. Single Deconvolutional Layer 1

15. Single Deconvolutional Layer 1

16. Single Deconvolutional Layer 1 Top-down Decomposition

17. Single Deconvolutional Layer 1

18. Single Deconvolutional Layer 1

19. Toy Example Featuremaps Filters

20. Objective for Single Layer • = Input, = Feature maps, = Filters • Simplify Notation:: • Filters are parameters of model (shared across all images) • Feature maps are latent variables (specific to an image) min z:

21. Inference for Single Layer Objective: • Known: = Input, = Filter weights. Solve for : = Feature maps • Iterative Shrinkage & Thresholding Algorithm (ISTA) Alternate: • 1) Gradient step: • 2) Shrinkage (per element): • Only parameter is ( can be automatically selected)

22. Effect of Sparsity • Introduces local competition in feature maps • Explaining away • Implicit grouping mechanism • Filters capture commonstructures • Thus only a single dot in feature maps is needed to reconstruct large structures

23. Talk Overview • Single layer • Convolutional Sparse Coding • Max Pooling • Multiple layers • Multi-layer inference • Filter learning • Comparison to related methods • Experiments

24. Reversible Max Pooling Pooled Feature Maps MaxLocations “Switches” Pooling Unpooling Feature Map Reconstructed Feature Map

25. 3D Max Pooling • Pool within & between feature maps • Take absolute max value (& preserve sign) • Record locations of max in switches

26. Role of Switches • Permit reconstruction path back to input • Record position of local max • Important for multi-layer inference • Set during inference of each layer • Held fixed for subsequent layers’ inference • Provide invariance: • Single feature map

27. Overall Architecture (1 layer)

28. Toy Example Pooledmaps Featuremaps Filters

29. Effect of Pooling • Reduces size of feature maps • So we can have more of them in layers above • Pooled maps are dense • Ready to be decomposed by sparse coding of layer above • Benefits of 3D Pooling • Added Competition • Local L0 Sparsity • AND/OR Effect

30. Talk Overview • Single layer • Convolutional Sparse Coding • Max Pooling • Multiple layers • Multi-layer inference • Filter learning • Comparison to related methods • Experiments

31. Stacking the Layers • Take pooled maps as input to next deconvolution/pooling layer • Learning & inference is layer-by-layer • Objective is reconstruction error • Key point: with respect to input image • Constraint of using filters in layers below • Sparsity & pooling make model non-linear • No sigmoid-type non-linearities

32. Overall Architecture (2 layers)

33. Multi-layer Inference • Consider layer 2 inference: • Want to minimize reconstruction error ofinput image , subject to sparsity. • Don’t care about reconstructing layers below • ISTA: • Update : • Shrink : • Update switches, : • No explicit non-linearities between layers • But still get very non-linear behavior

34. Filter Learning • Update Filters with Conjugate Gradients: • For Layer 1: • For higher layers: • Obtain gradient by reconstructing down to image and projecting error back up to current layer • Normalize filters to be unit length Objective: • Known: = Input, = Feature maps. Solve for : = Filter weights

35. Overall Algorithm • For Layer 1 to L: % Train each layer in turn • For Epoch 1 to E: % Loops through dataset • For Image 1 to N: % Loop over images • For ISTA_step 1 to T: % ISTA iterations • Reconstruct % Gradient • Compute error % Gradient • Propagate error % Gradient • Gradient step % Gradient • Skrink% Shrinkage • Pool/Update Switches % Update Switches • Update filters % Learning, via linear CG system

36. 2nd layer pooled maps 2nd layer feature maps 2nd layer filters 1st layer pooled maps 1st layer feature maps 1st layer filters Toy Input

37. Talk Overview • Single layer • Convolutional Sparse Coding • Max Pooling • Multiple layers • Multi-layer inference • Filter learning • Comparison to related methods • Experiments

38. Related Work • Convolutional Sparse Coding • Zeiler, Krishnan, Taylor & Fergus [CVPR ’10] • Kavukcuoglu, Sermanet, Boureau, Gregor, Mathieu & LeCun [NIPS ’10] • Chen, Spario, Dunson & Carin [JMLR submitted] • Only 2 layer models • Deep Learning • Hinton & Salakhutdinov [Science ‘06] • Ranzato, Poultney, Chopra & LeCun [NIPS ‘06] • Bengio, Lamblin, Popovici & Larochelle [NIPS ‘05] • Vincent, Larochelle, Bengio & Manzagol [ICML ‘08] • Lee, Grosse, Ranganth & Ng [ICML ‘09] • Jarrett, Kavukcuoglu, Ranzato & LeCun [ICCV ‘09] • Ranzato, Mnih, Hinton [CVPR’11] • Reconstruct layer below, not input • Deep Boltzmann Machines • Salakhutdinov & Hinton [AIStats’09]

39. Comparison: Convolutional Nets LeCunet al. 1989 • Deconvolutional Networks • Top-down decomposition with convolutions in feature space. • Non-trivial unsupervised optimization procedure involving sparsity. • Convolutional Networks • Bottom-up filtering with convolutions in image space. • Trained supervised requiring labeled data.

40. Related Work • Hierarchical vision models • Zhu & Mumford [F&T ‘06] • Tu & Zhu [IJCV ‘06] • Serre, Wolf & Poggio [CVPR ‘05] Fidler & Leonardis [CVPR ’07] • Zhu & Yuille [NIPS ’07] • Jin & Geman [CVPR ’06]

41. Talk Overview • Single layer • Convolutional Sparse Coding • Max Pooling • Multiple layers • Multi-layer inference • Filter learning • Comparison to related methods • Experiments

42. Training Details • 3060 training images from Caltech 101 • 30 images/class, 102 classes (Caltech 101 training set) • Resized/padded to 150x150 grayscale • Subtractive & divisive contrast normalization • Unsupervised • 6 hrs total training time (Matlab, 6 core CPU)

43. Model Parameters/Statistics 7x7 filters at all layers

44. Model Reconstructions

45. Layer 1 Filters • 15 filters/feature maps, showing max for each map

46. Visualization of Filters from Higher Layers • Raw coefficients are difficult to interpret • Don’t show effect of switches • Take max activation from feature map associated with each filter • Project back to input image (pixel space) • Use switches in lower layers peculiar to that activation FeatureMap .... Filters Lower Layers Input Image Training Images

47. Layer 2 Filters • 50 filters/feature maps, showing max for each mapprojected down to image

48. Layer 3 filters • 100 filters/feature maps, showing max for each map

49. Layer 4 filters • 150 in total; receptive field is entire image

50. Relative Size of Receptive Fields (to scale)