1 / 85

Petacat : Applying ideas from Copycat to image understanding

Petacat : Applying ideas from Copycat to image understanding. How Streetscenes Works ( Bileschi , 2006). 1. Densely tile the image with windows of different sizes. 2. HMAX C2 features are computed in each window. 3. The features in each window are given as input

kass
Download Presentation

Petacat : Applying ideas from Copycat to image understanding

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Petacat: Applying ideas from Copycat to image understanding

  2. How Streetscenes Works(Bileschi, 2006) 1. Densely tile the image with windows of different sizes. 2. HMAX C2 features are computed in each window. 3. The features in each window are given as input to each of five trained support vector machines (“pedestrian”, “car”, “bicycle”, “building”, “tree”) 4. If any return a classification with score above a learned threshold, that object is said to be “detected” . …

  3. Object detection (here, “car”) with HMAX model (Bileschi, 2006)

  4. Limitations of Streetscenes approach for “image understanding”

  5. Limitations of Streetscenes approach for “image understanding” • Exhaustive search – not scalable • Does not recognize spatial and abstract relationships among objects for whole scene understanding • Has no prior knowledge about object categories and their place in “conceptual space” • HMAX model is completely feed-forward; no feedback to allow context to aid in scene understanding. • Where should feedback come in?

  6. Representation of High-Level Knowledge: A Simple Semantic Network (or “Ontology”) “Dog walking” Person Dog leash holds attached to action action walking

  7. But...

  8. Modified Ontology “Dog walking” Person Dog leash holds attached to Dog Group action action running walking

  9. Modified Ontology “Dog walking” Person Dog leash holds attached to Dog Group action action Allowing “conceptual slippage” running walking

  10. But...

  11. Modified Ontology “Dog walking” holds attached to leash Dog Group Person Dog action Cat action walking running Iguana

  12. But...

  13. But...

  14. But...

  15. But...

  16. Modified Ontology “Dog walking” Person Dog leash Helicopter Bicycle Car holds attached to Dog Group action action Cat running Iguana walking

  17. But...

  18. Lawn mower Attached to Helicopter Gasoline Fanny pack Sidewalk Beach Stick Inside Runway Sky Leash Army Grass Airplane Ground Outside Dog Person Dog walking Dog grooming Holding Tree Backpack Standing Close to Above Walking Car Running Track Left of Far from

  19. Need dynamical process of constructing representation.

  20. Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes.

  21. Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. • Ongoing perception of “context” brings in appropriate concepts and conceptual slippages, and avoids exhaustive search

  22. Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. • Ongoing perception of “context” brings in appropriate concepts and conceptual slippages, and avoids exhaustive search • Prior, higher-level knowledge interacts with lower-level vision in both directions (bottom-up and top-down).

  23. Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. • Ongoing perception of “context” brings in appropriate concepts and conceptual slippages, and avoids exhaustive search • Prior, higher-level knowledge interacts with lower-level vision in both directions (bottom-up and top-down). • Concepts are “fluid”, allowed to “slip” in certain contexts.

  24. Need dynamical process of constructing representation. Information gained during the unfolding of perception feeds back to guide the directions the perceptual process takes. • Ongoing perception of “context” brings in appropriate concepts and conceptual slippages, and avoids exhaustive search • Prior, higher-level knowledge interacts with lower-level vision in both directions (bottom-up and top-down). • Concepts are “fluid”, allowed to “slip” in certain contexts. • This allows perception of essential similarity in the face of superficial differences—i.e., analogy-making.

  25. Active Symbol Architecture(Hofstadter et al., 1995)

  26. Active Symbol Architecture(Hofstadter et al., 1995) • Basis for • Copycat (analogy-making), Hofstadter & Mitchell • Tabletop (anlaogy-making), Hofstadter & French • Metacat(analogy-making and self-awareness), Hofstadter & Marshall and many others…

  27. Semantic network Active Symbol Architecture(Hofstadter et al., 1995) Workspace Temperature Perceptual agents (codelets)

  28. Petacat:(Descendant of Copycat)Integration of Active Symbol Architecture and HMAX Initial task: Decide if image is an instance of “taking a dog for a walk”, and if so, how good an instance it is.

  29. Semantic Network indoors taking a dog for a walk has location Object outdoors has component has component has component Action grass is in front of a sidewalk beach a person Spatial Relation dog is on is on is touching leash is touching is on road a has action is behind is next to has action horse cat has component trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits

  30. Semantic Network indoors Property links taking a dog for a walk has location Object outdoors Slip links has component has component has component Action grass is in front of a sidewalk beach a person Spatial Relation dog is on is on is touching leash is touching is on road a has action is next to is behind has action horse cat has component trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits

  31. Semantic Network indoors Property links taking a dog for a walk has location Object outdoors Slip links has component has component has component Action grass is in front of a sidewalk beach a person Spatial Relation dog is on is on is touching leash is touching is on road a has action is next to is behind has action horse cat has component trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits Properties of nodes

  32. Workspace

  33. Semantic network Workspace

  34. Semantic network Perceptual Agents (Codelets) Codelets as active symbols

  35. indoors taking a dog for a walk has location Object outdoors has component has component has component Action is in front of grass a beach a person is on dog is on is touching Spatial Relation leash is touching is on road a sidewalk has action is behind is next to has action horse cat has component trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits

  36. indoors taking a dog for a walk has location Object outdoors has component has component has component Action is in front of grass a beach a is on person dog is on is touching Spatial Relation leash is touching is on road a sidewalk has action is behind is next to has action horse has component cat trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits

  37. indoors taking a dog for a walk has location outdoors Object has component has component has component Action is in front of grass a beach a is on person dog is on Spatial Relation is touching is on sidewalk leash is touching has action road a is behind is next to has component has action horse cat trail belt walks walks rope is in front of has location is touching runs flies string drives stands swims sits

  38. indoors taking a dog for a walk has location Object outdoors has component has component has component Action is in front of grass a beach a is on person dog is on is touching Spatial Relation leash is touching is on road a sidewalk has action is behind is next to has action horse cat has component trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits

  39. Illustration of what we plan to have happen – not a real run of Petacat Dog?

  40. Illustration of what we plan to have happen – not a real run of Petacat Person? Dog? Dog?

  41. Illustration of what we plan to have happen – not a real run of Petacat Person? Dog? Dog? Sidewalk?

  42. Illustration of what we plan to have happen – not a real run of Petacat Outdoors? Person? Dog? Dog? Dog? Sidewalk?

  43. Illustration of what we plan to have happen – not a real run of Petacat Outdoors? Person? Dog? Dog? Dog? Sidewalk? Scout codelets: Send C1 features in window to corresponding SVM. If positive result, post builder codeletwith urgency equal to SVM’s confidence.

  44. Illustration of what we plan to have happen – not a real run of Petacat Outdoors? positive: 0.7 Person? negative Dog? negative Dog? negative Dog? positive: 0.8 Sidewalk? positive: 0.4 Scout codelets: Send C1 features in window to corresponding SVM. If positive result, post builder codeletwith urgency equal to SVM’s confidence.

  45. Illustration of what we plan to have happen – not a real run of Petacat Outdoors? positive: 0.7 Person? negative Dog? negative Dog? negative Dog? positive: 0.8 Sidewalk? positive: 0.4 Builder codelets: Ask HMAX to compute C2 features using prototypes specific to the object (or scene), and send them to corresponding SVM. If positive, decide to build structure with probability equal to SVM confidence. Break competing structures if necessary.

  46. Illustration of what we plan to have happen – not a real run of Petacat Outdoors Dog Builder codelets: Ask HMAX to compute object-/scene-specific C2 features, and send them to corresponding SVM. If positive, decide to build structure with probability equal to SVM confidence. Break competing structures if necessary.

  47. indoors taking a dog for a walk has location Object outdoors has component has component has component Action is in front of grass a beach a is on person dog is on is touching Spatial Relation leash is touching is on road a sidewalk has action is behind is next to has action horse cat has component trail belt walks walks rope is in front of runs has location is touching flies string drives stands swims sits

More Related