1 / 58

Large-Scale Image Parsing

sky. building. car. Large-Scale Image Parsing. road. Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill. Small-scale image parsing Tens of classes, hundreds of images. Figure from Shotton et al. (2009).

alagan
Download Presentation

Large-Scale Image Parsing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. sky building car Large-Scale Image Parsing road Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill

  2. Small-scale image parsingTens of classes, hundreds of images Figure from Shotton et al. (2009) He et al. (2004), Hoiem et al. (2005), Shotton et al. (2006, 2008, 2009), Verbeek and Triggs (2007), Rabinovich et al. (2007), Galleguillos et al. (2008), Gould et al. (2009), etc.

  3. Large-scale image parsingHundreds of classes, tens of thousands of images Non-uniform class frequencies

  4. Large-scale image parsingHundreds of classes, tens of thousands of images Non-uniform class frequencies Evolving training set http://labelme.csail.mit.edu/

  5. Challenges • What’s considered important for small-scale image parsing? • Combination of local cues • Multiple segmentations, multiple scales • Context • How much of this is feasible for large-scale, dynamic datasets?

  6. Our first attempt: A nonparametric approach • Lazy learning: do (almost) nothing up front • To parse (label) an image we will: • Find a set of similar images • Transfer labels from the similar images by matching pieces of the image (superpixels)

  7. Finding Similar Images

  8. Ocean Forest Open Field Mountain Which image is most similar? Highway Inner City Then assign the label from the most similar image Street Tall Building What is depicted in this image?

  9. Pixels are a bad measure of similarity Most similar according to pixel distance Most similar according to “Bag of Words”

  10. Origin of the Bag of Words model • Orderless document representation: • frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloudhttp://chir.ag/phernalia/preztags/

  11. What are words for an image?

  12. Wing Tail Propeller Building Wheel

  13. Wing Building Wheel Propeller Jet Engine Tail

  14. Wing Building Wheel Propeller Jet Engine Tail

  15. Wing Building Wheel Propeller Jet Engine Tail

  16. But where do the words come from?

  17. Then where does the dictionary come from?

  18. Example Dictionary Source: B. Leibe

  19. … … … Another dictionary Source: B. Leibe

  20. Fei-Fei et al. 2005

  21. Outline of the Bag of Words method • Divide the image into patches • Assign a “word” for each patch • Count the number of occurrences of each “word” in the image

  22. Does this work for our problem? 65,536 Pixels 256 Dimensions

  23. Which look the most similar?

  24. Which look the most similar? sky sky building tree building tree car car road road sky sky sky building building building sand car road sky sky sky sky building building mountain car car tree car road road road

  25. Step 1: Scene-level matching Gist (Oliva & Torralba, 2001) Spatial Pyramid(Lazebnik et al., 2006) Color Histogram Retrieval set: Source of possible labels Source of region-level matches

  26. Step 2: Region-level matching

  27. Step 2: Region-level matching Superpixels(Felzenszwalb & Huttenlocher, 2004)

  28. Step 2: Region-level matching Road Tree Pixel Area (size) Sky Building Snow

  29. Step 2: Region-level matching Road Absolute mask(location) Sidewalk

  30. Step 2: Region-level matching Road Texture Snow Sky Sidewalk

  31. Step 2: Region-level matching Road Sidewalk Color histogram Building

  32. Step 2: Region-level matching Superpixel features Superpixels(Felzenszwalb & Huttenlocher, 2004)

  33. Region-level likelihoods • Nonparametric estimate of class-conditional densities for each class c and feature type k: • Per-feature likelihoods combined via Naïve Bayes: Features of class c within some radius of ri Total features of class c in the dataset kth feature type of ith region

  34. Region-level likelihoods Building Car Crosswalk Road Window Sky

  35. Step 3: Global image labeling • How do we resolve issues like this? Maximum likelihood labeling Original image sky sky road tree sea sea road sand sand

  36. Step 3: Global image labeling • Compute a global image labeling by optimizing a Markov random field (MRF) energy function: Likelihood score for region ri and label ci Smoothing penalty Co-occurrence penalty Neighboring regions Regions Vector of region labels

  37. Step 3: Global image labeling • Compute a global image labeling by optimizing a Markov random field (MRF) energy function: Likelihood score for region ri and label ci Smoothing penalty Co-occurrence penalty Neighboring regions Regions Vector of region labels Maximum likelihood labeling Edge penalties Final labeling Final edge penalties sky sky building building window car car road road

  38. Step 3: Global image labeling • Compute a global image labeling by optimizing a Markov random field (MRF) energy function: Likelihood score for region ri and label ci Smoothing penalty Co-occurrence penalty Neighboring regions Regions Vector of region labels Maximum likelihood labeling Edge penalties Original image MRF labeling sky sky road tree sea sea road sand sand

  39. Joint geometric/semantic labeling • Semantic labels: road, grass, building, car, etc. • Geometric labels: sky, vertical, horizontal • Gould et al. (ICCV 2009) Original image Semantic labeling Geometric labeling sky sky tree vertical car horizontal road

  40. Joint geometric/semantic labeling • Objective function for joint labeling: Geometric labels Semantic labels Cost of geometric labeling Cost of semantic labeling Geometric/semantic consistency penalty Original image Semantic labeling Geometric labeling sky sky tree vertical car horizontal road

  41. Example of joint labeling

  42. Understanding scenes on many levels To appear at ICCV 2011

  43. Datasets

  44. Datasets

  45. Overall performance *SIFT Flow: 74.75

More Related