1 / 63

Roles of Appearance and Contextual Information

Roles of Appearance and Contextual Information. Devi Parikh , Larry Zitnick and Tsuhan Chen. When?. High intra-class variance Chairs Low inter-class variance Lemon vs. tennis ball Occlusion Low resolution image Image of a far away scene Bad quality. [Bustos et.al., CSUR 2005].

elvin
Download Presentation

Roles of Appearance and Contextual Information

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Roles of Appearance and Contextual Information Devi Parikh, Larry Zitnick and Tsuhan Chen

  2. When? • High intra-class variance • Chairs • Low inter-class variance • Lemon vs. tennis ball • Occlusion • Low resolution image • Image of a far away scene • Bad quality [Bustos et.al., CSUR 2005] [Rabinovich et al., ICCV 07]

  3. Popular motivations for context… [Antonio Torralba]

  4. Popular motivations for context… [Antonio Torralba]

  5. Popular motivations for context… [Antonio Torralba]

  6. Popular motivations for context… [Antonio Torralba]

  7. However… Let’s look at it more carefully…

  8. Object appearance

  9. Appearance + Context

  10. Blind recognition

  11. Low resolution

  12. High resolution

  13. Blind recognition

  14. Appearance + Context

  15. When? • Humans? • Machines?

  16. Set-up ? ? ? ? ? Appearance Machine Low Resolution Context High Resolution Human

  17. Low resolution: Appearance

  18. Low resolution: Context

  19. Low resolution: Appearance + Context

  20. High resolution: Appearance

  21. High resolution: Context

  22. High resolution: Appearance + Context 11 subjects 2 sessions 3 scenarios 70 segments

  23. Related work [Torralba et al., Tech Report 2007]

  24. Machines ? ? ? ? ? Inference: Belief Propagation Appearance Context Color [GMM] Texture + Shape [TextonBoost]* Co-occurrence** Relative location Relative scale Neural Network **[Rabinovich et al., ICCV 2007] *[Shotton et al., ECCV 2006]

  25. Machines Relative location Co-occurrence** Relative scale **[Rabinovich2007]

  26. Machines • MSRC dataset • Corel dataset [Felzenshwalb and Huttenlocher IJCV 2004]

  27. Results

  28. Results Low High High resolution images do not benefit from context A: Appearance alone C: Context alone A+C: Appearance and context

  29. Results Low High Low resolution images NEED context A: Appearance alone C: Context alone A+C: Appearance and context

  30. Results

  31. Results (Machine) Context & Appearance help each other

  32. Results (Machine) Appearance hurts

  33. Results (Machine) Context very weak

  34. Results (Machine) Context makes no difference

  35. Results (Machine)

  36. Results (Machine) Co-occurrence Relative Location Relative Scale

  37. Results (Machine) Failure cases

  38. Results (Machine) 1 [Yang et al., CVPR 2007] 2 [He et al., ECCV 2006]

  39. Results (Machine)

  40. Contributions • Context is most useful when appearance information is weak • Low resolution images are an appropriate venue for studying context

  41. Discussion • Improve appearance or context models to achieve human performance? • Need to improve both appearance and context models • In low resolution images, appearance information is similar for humans and machines • Hence, appropriate venue for studying context • Achieving human performances need not be the ultimate goal Need to improve appearance models Need to improve context models

  42. Follow up work (PAMI 2011)

  43. Machine do not leverage contextual information as effectively as humans Results Low High A: Appearance alone C: Context alone A+C: Appearance and context Are machine missing a source of context?

  44. Different Sources of Context

  45. Different Sources of Context: None

  46. Different Sources of Context: Cooc

  47. Different Sources of Context: Rel-scale

  48. Different Sources of Context: Rel-loc

  49. Different Sources of Context: All

  50. Different Sources of Context: All

More Related