1 / 47

General architecture

General architecture. Minimal Subscene. Working definition: The smallest set of objects, actors and actions in a dynamic visual scene that are relevant to present behavior For now we will assume: Bottom-up: objects/actors/actions must be visible

donalddunn
Download Presentation

General architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Generalarchitecture CS 664, Session 19

  2. Minimal Subscene • Working definition: The smallest set of objects, actors and actions in a dynamic visual scene that are relevant to present behavior For now we will assume: • Bottom-up: objects/actors/actions must be visible • Top-down: relevance to present behavior explicitly specified, e.g., by specifying a question or task • Knowledge base: the system may supplement explicit knowledge with long-term acquired knowledge CS 664, Session 19

  3. Motivation:Humans • 1) Free examination • 2) estimate material • circumstances of family • 3) give ages of the people • 4) surmise what family has • been doing before arrival • of “unexpected visitor” • 5) remember clothes worn by • the people • 6) remember position of people • and objects • 7) estimate how long the “unexpected • visitor” has been away from family CS 664, Session 19 Yarbus, 1967

  4. “Beobot” CS 664, Session 19

  5. VisualAttention see http://iLab.usc.edu CS 664, Session 19

  6. ObjectRecognition Riesenhuber & Poggio, Nat Neurosci, 1999 (MIT) CS 664, Session 19

  7. Action Recognition Oztop & Arbib, 2001 CS 664, Session 19

  8. Start: • Issue question • Parse question • Extract keywords • Expand to related concepts, • using ontology/KB • -Fill initial “task list” CS 664, Session 19

  9. Task list Working list of currently relevant objects/actors/actions • Initially empty • Question/task specification provides initial filling-in • As the scene is scanned and objects/actors/actions are recognized, contents of task list are updated CS 664, Session 19

  10. “Where:” attention, saliency map and task map Input: video stream Low-level vision: massively parallel extraction of simple visual features from video input Saliency map: localizes conspicuous (potentially interesting) objects irrespectively of why they are salient Task map: acts as spatial filter to saliency map; only locations in the current minimal subscene can easily pass through. Other locations need to be exceptionally salient to pass through. CS 664, Session 19

  11. “What” memory Relates concepts to visual properties Bridge between visual and semantic knowledge CS 664, Session 19

  12. Generalarchitecture CS 664, Session 19

  13. Examples / experiments • Examine video clips • For each scene, please write down: • Most salient object • Most salient action • Minimal subscene • Who is doing what to whom CS 664, Session 19

  14. Scene 001 CS 664, Session 19

  15. Scene 001 – Attentional Trajectory CS 664, Session 19

  16. Scene 002 CS 664, Session 19

  17. Scene 002 – Attentional Trajectory CS 664, Session 19

  18. Scene 003 CS 664, Session 19

  19. Scene 003 – Attentional Trajectory CS 664, Session 19

  20. Scene 004 CS 664, Session 19

  21. Scene 004 – Attentional Trajectory CS 664, Session 19

  22. Scene 005 CS 664, Session 19

  23. Scene 005 – Attentional Trajectory CS 664, Session 19

  24. Scene 006 CS 664, Session 19

  25. Scene 006 – Attentional Trajectory CS 664, Session 19

  26. Scene 007 CS 664, Session 19

  27. Scene 007 – Attentional Trajectory CS 664, Session 19

  28. Scene 008 CS 664, Session 19

  29. Scene 008 – Attentional Trajectory CS 664, Session 19

  30. Scene 009 CS 664, Session 19

  31. Scene 009 – Attentional Trajectory CS 664, Session 19

  32. Scene 010 CS 664, Session 19

  33. Scene 010 – Attentional Trajectory CS 664, Session 19

  34. Scene 011 CS 664, Session 19

  35. Scene 011 – Attentional Trajectory CS 664, Session 19

  36. Scene 012 CS 664, Session 19

  37. Scene 012 – Attentional Trajectory CS 664, Session 19

  38. Scene 013 CS 664, Session 19

  39. Scene 013 – Attentional Trajectory CS 664, Session 19

  40. Scene 014 CS 664, Session 19

  41. Scene 014 – Attentional Trajectory CS 664, Session 19

  42. Scene 015 CS 664, Session 19

  43. Scene 015 – Attentional Trajectory CS 664, Session 19

  44. Scene 016 CS 664, Session 19

  45. Scene 016 – Attentional Trajectory CS 664, Session 19

  46. Scene 017 CS 664, Session 19

  47. Scene 017 – Attentional Trajectory CS 664, Session 19

More Related