1 / 48

Learning sparse representations to restore, classify, and sense images and videos

Learning sparse representations to restore, classify, and sense images and videos. Guillermo Sapiro University of Minnesota. Supported by NSF, NGA, NIH, ONR, DARPA, ARO, McKnight Foundation. Ramirez. Martin Duarte. Lecumberry. Rodriguez. Overview. Introduction

Download Presentation

Learning sparse representations to restore, classify, and sense images and videos

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Learning sparse representationsto restore, classify, and sense images and videos Guillermo Sapiro University of Minnesota Supported by NSF, NGA, NIH, ONR, DARPA, ARO, McKnight Foundation

  2. Ramirez Martin Duarte Lecumberry Rodriguez Learning Sparsity

  3. Overview • Introduction • Denoising, Demosaicing, Inpainting • Mairal, Elad, Sapiro, IEEE-TIP, January 2008 • Learn multiscale dictionaries • Mairal, Elad, Sapiro, SIAM-MMS, April 2008 • Sparsity + Self-similarity • Mairal, Bach, Ponce, Sapiro, Zisserman, pre-print. • Incoherent dictionaries and universal coding • Ramirez, Lecumberry, Sapiro, June 2009, pre-print • Learning to classify • Mairal, Bach, Ponce, Sapiro, Zisserman, CVPR 2008, NIPS 2008 • Rodriguez and Sapiro, pre-print, 2008. • Learning to sense sparse signals • Duarte and Sapiro, pre-print, May 2008, IEEE-TIP to appear Learning Sparsity

  4. Introduction I: Sparse and Redundant Representations Webster Dictionary: Of few and scattered elements Learning Sparsity

  5. Relation to measurements Prior or regularization ThomasBayes 1702 - 1761 Restoration by Energy Minimization Restoration/representation algorithms are often related to the minimization of an energy function of the form y : Given measurements x : Unknown to be recovered • Bayesian type of approach • What is the prior? What is the image model? Learning Sparsity

  6. Every column in D (dictionary) is a prototype signal (Atom). N • The vector  contains very few (say L) non-zeros. N A sparse & random vector K A fixed Dictionary The Sparseland Model for Images M Learning Sparsity

  7. D should be chosen such that it sparsifies the representations (for a given task!) Learn D : Multiscale Learning Color Image Examples Task / sensing adapted Internal structure One approach to choose D is from a known set of transforms (Steerable wavelet, Curvelet, Contourlets, Bandlets, …) What Should the Dictionary D Be? Learning Sparsity

  8. Introduction II: Dictionary Learning Learning Sparsity

  9. D X  A Each example has a sparse representation with no more than L atoms Each example is a linear combination of atoms from D Measure of Quality for D Field & Olshausen (‘96) Engan et. al. (‘99) Lewicki & Sejnowski (‘00) Cotter et. al. (‘03) Gribonval et. al. (‘04) Aharon, Elad, & Bruckstein (‘04) Aharon, Elad, & Bruckstein (‘05) Ng et al. (‘07) Mairal, Sapiro, Elad (‘08) Learning Sparsity

  10. D Initialize D Sparse Coding Orthogonal Matching Pursuit (or L1) XT Dictionary Update Column-by-Column by SVD computation over the relevant examples The K–SVD Algorithm – General Aharon, Elad, & Bruckstein (`04) Learning Sparsity

  11. Show me the pictures Learning Sparsity

  12. Change the Metric in the OMP Learning Sparsity

  13. Non-uniform noise Learning Sparsity

  14. Example: Non-uniform noise Learning Sparsity

  15. Example: Inpainting Learning Sparsity

  16. Example: Demoisaic Learning Sparsity

  17. Example: Inpainting Learning Sparsity

  18. Not enough fun yet?: Multiscale Dictionaries Learning Sparsity

  19. Learned multiscale dictionary Learning Sparsity

  20. Learning Sparsity

  21. Color multiscale dictionaries Learning Sparsity

  22. Example Learning Sparsity

  23. Video inpainting Learning Sparsity

  24. Extending the Models Learning Sparsity

  25. Universal Coding and Incoherent Dictionaries • Consistent • Improved generalization properties • Improved active set computation • Improved coding speed • Improved reconstruction • See poster by Ramirez and Lecumberry… Learning Sparsity

  26. Sparsity + Self-similarity=Group Sparsity • Combine the two of the most successful models for images • Mairal, Bach. Ponce, Sapiro, Zisserman, pre-print, 2009 Learning Sparsity

  27. Learning to Classify

  28. Global Dictionary Learning Sparsity

  29. Barbara Learning Sparsity

  30. Boat Learning Sparsity

  31. Digits Learning Sparsity

  32. Which dictionary? How to learn them? • Multiple reconstructive dictionary? (Payre) • Single reconstructive dictionary? (Ng et al, LeCunn et al.) • Dictionaries for classification! • See also Winn et al., Holub et al., Lasserre et al., Hinton et al. for joint discriminative/generative probabilistic approaches Learning Sparsity

  33. Learning multiple reconstructive and discriminative dictionaries With J. Mairal, F. Bach, J. Ponce, and A. Zisserman, CVPR ’08, NIPS ‘08 Learning Sparsity

  34. Texture classification Learning Sparsity

  35. Semi-supervised detection learning MIT -- Learning Sparsity

  36. Learning a Single Discriminative and Reconstructive Dictionary • Exploit the representation coefficients for classification • Include this in the optimization • Class supervised simultaneous OMP With F. Rodriguez Learning Sparsity

  37. Digits images: Robust to noise and occlusions Learning Sparsity

  38. Supervised Dictionary Learning With J. Mairal, F. Bach, J. Ponce, and A. Zisserman, NIPS ‘08 Learning Sparsity

  39. Learning to Sense Sparse Images

  40. Motivation • Compressed sensing (Candes &Tao, Donoho, et al.) • Sparsity • Random sampling • Universality • Stability • Shall the sensing be adapted to the data type? • Yes! (Elad, Peyre, Weiss et al., Applebaum et al, this talk). • Shall the sensing and dictionary be learned simultaneously? Learning Sparsity

  41. Some formulas…. + “RIP (Identity Gramm Matrix)” Learning Sparsity

  42. Design the dictionary and sensing together Learning Sparsity

  43. Just Believe the Pictures Learning Sparsity

  44. Just Believe the Pictures Learning Sparsity

  45. Just Believe the Pictures Learning Sparsity

  46. Conclusions • State-of-the-art denoising results for still (shared with Dabov et al.) and video • General • Vectorial and multiscale learned dictionaries • Dictionaries with internal structure • Dictionary learning for classification • See also Szlam and Sapiro, ICML 2009 • See also Carin et al, ICML 2009 • Dictionary learning for sensing • A lot of work still to be done! Learning Sparsity

  47. Please do not use the wrong dictionaries… • 12 M pixel image • 7 million patches • LARS+online learning: ~8 minutes • Mairal, Bach, Ponce, Sapiro, ICML 2009 Learning Sparsity

  48. Learning Sparsity

More Related