1 / 71

Computational Photography

Computational Photography. Tamara Berg Features. Representing Images. Keep all the pixels!. Pros? Cons?. 2 nd Try. Compute average pixel. Pros? Cons?. 3rd Try. Photo by: marielito. Represent the image as a spatial grid of average pixel colors Pros? Cons?. QBIC system.

taite
Download Presentation

Computational Photography

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computational Photography Tamara Berg Features

  2. Representing Images Keep all the pixels! Pros? Cons?

  3. 2nd Try Compute average pixel Pros? Cons?

  4. 3rdTry Photo by: marielito Represent the image as a spatial grid of average pixel colors Pros? Cons?

  5. QBIC system • First content based image retrieval system • Query by image content (QBIC) • IBM 1995 • QBIC interprets the virtual canvas as a grid of coloured areas, then matches this grid to other images stored in the database. QBIC link

  6. Feature Choices

  7. Global vs Local? Depends whether you want whole or local image similarity Beach Person

  8. Feature types! The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different!

  9. Outline Image Features Global vsLocal – depends on the task Feature Types: color histograms (color), texture histograms (texture), SIFT (shape)

  10. Global Features • The “gist” of a scene: Oliva & Torralba (2001) source: Svetlana Lazebnik

  11. Scene Descriptor Compute oriented edge response at multiple scales (5 spatial scales, 6 orientations) Hays and Efros, SIGGRAPH 2007

  12. Scene Descriptor Gist scene descriptor (Oliva and Torralba 2001) “semantic” descriptor of image composition aggregated edge responses over 4x4 windows scenes tend to be semantically similar under this descriptor if very close Hays and Efros, SIGGRAPH 2007

  13. Local Features Feature points (locations) + feature descriptors source: Svetlana Lazebnik

  14. Why extract features? • Motivation: panorama stitching • We have two images – how do we combine them? source: Svetlana Lazebnik

  15. Step 2: match features Why extract features? • Motivation: panorama stitching • We have two images – how do we combine them? Step 1: extract features source: Svetlana Lazebnik

  16. Why extract features? • Motivation: panorama stitching • We have two images – how do we combine them? Step 1: extract features Step 2: match features Step 3: align images source: Svetlana Lazebnik

  17. Characteristics of good features • Repeatability • The same feature can be found in several images despite geometric and photometric transformations • Saliency • Each feature has a distinctive description • Compactness and efficiency • Many fewer features than image pixels • Locality • A feature occupies a relatively small area of the image; robust to clutter and occlusion source: Svetlana Lazebnik

  18. Applications • Feature points are used for: • Retrieval • Indexing • Motion tracking • Image alignment • 3D reconstruction • Object recognition • Indexing and database retrieval • Robot navigation source: Svetlana Lazebnik

  19. Feature Types Shape! Color! Texture!

  20. Color Features

  21. RGB Colorspace • Primaries are monochromatic lights (for monitors, they correspond to the three types of phosphors) RGB matching functions source: Svetlana Lazebnik

  22. HSV Colorspace • Perceptually meaningful dimensions: Hue, Saturation, Value (Intensity) source: Svetlana Lazebnik

  23. Color Histograms

  24. Uses of color in computer vision Color histograms for indexing and retrieval Swain and Ballard, Color Indexing, IJCV 1991. source: Svetlana Lazebnik

  25. Uses of color in computer vision Skin detection M. Jones and J. Rehg, Statistical Color Models with Application to Skin Detection, IJCV 2002. source: Svetlana Lazebnik

  26. Uses of color in computer vision Image segmentation and retrieval C. Carson, S. Belongie, H. Greenspan, and Ji. Malik, Blobworld: Image segmentation using Expectation-Maximization and its application to image querying, ICVIS 1999. source: Svetlana Lazebnik

  27. Color features Pros/Cons?

  28. Interaction of light and surfaces • Reflected color is the result of interaction of light source spectrum with surface reflectance source: Svetlana Lazebnik

  29. Interaction of light and surfaces Olafur Eliasson, Room for one color source: Svetlana Lazebnik

  30. Texture Features

  31. Texture Features • Texture is characterized by the repetition of basic elements or textons • For stochastic textures, it is the identity of the textons, not their spatial arrangement, that matters Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

  32. Texture Histograms histogram Universal texton dictionary Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

  33. Shape Features

  34. We want invariance!!! • Good features should be robust to all sorts of nastiness that can occur between images. Slide source: Tom Duerig

  35. Types of invariance • Illumination Slide source: Tom Duerig

  36. Types of invariance • Illumination • Scale Slide source: Tom Duerig

  37. Types of invariance • Illumination • Scale • Rotation Slide source: Tom Duerig

  38. Types of invariance • Illumination • Scale • Rotation • Affine Slide source: Tom Duerig

  39. Edge detection • Goal: Identify sudden changes (discontinuities) in an image • Intuitively, most semantic and shape information from the image can be encoded in the edges • More compact than pixels • Ideal: artist’s line drawing (but artist is also using object-level knowledge) Source: D. Lowe

  40. Origin of edges • Edges are caused by a variety of factors: surface normal discontinuity depth discontinuity surface color discontinuity illumination discontinuity Source: Steve Seitz

  41. intensity function(along horizontal scanline) first derivative edges correspond toextrema of derivative Characterizing edges • An edge is a place of rapid change in the image intensity function image source: Svetlana Lazebnik

  42. Finite difference filters • Other approximations of derivative filters exist: Source: K. Grauman

  43. Finite differences: example • Which one is the gradient in the x-direction (resp. y-direction)? source: Svetlana Lazebnik

  44. How to achieve illumination invariance • Use edges instead of raw values Slide source: Tom Duerig

  45. Local Features Feature points (locations) + feature descriptors source: Svetlana Lazebnik

  46. Local Features Feature points (locations) + feature descriptors Where should we put features? source: Svetlana Lazebnik

  47. Local Features Feature points (locations) + feature descriptors How should we describe them? source: Svetlana Lazebnik

  48. Where to put features? Edges source: Svetlana Lazebnik

  49. Finding Corners • Key property: in the region around a corner, image gradient has two or more dominant directions • Corners are repeatable and distinctive C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“Proceedings of the 4th Alvey Vision Conference: pages 147--151.  source: Svetlana Lazebnik

  50. “flat” region:no change in all directions “edge”:no change along the edge direction “corner”:significant change in all directions Corner Detection: Basic Idea • We should easily recognize the point by looking through a small window • Shifting a window in anydirection should give a large change in intensity Source: A. Efros source: Svetlana Lazebnik

More Related