360 likes | 511 Views
Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic. Del Bimbo, E. Vicario and D. Zingoni IEEE Transactions on Knowledge and Data Engineering, 7(4), August 1995. Outline. Overview Spatial logic Region-based formulation Object-based formulation
E N D
Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic Del Bimbo, E. Vicario and D. Zingoni IEEE Transactions on Knowledge and Data Engineering, 7(4), August 1995.
Outline • Overview • Spatial logic • Region-based formulation • Object-based formulation • Temporal logic • Sequence Description using STL • A System for Image Sequence Retrieval
Overview:Spatio-temporal logic (STL) • A symbolic representation of spatio-temporal relationships between objects within image sequences. • Completely based on logic. • Two main components: • Spatial logic for individual scenes • Temporal logic for scene sequences
Spatial logic • Expresses geometric ordering and relationships between objects • In fact, between the projections of the objects in N-dimensional space. • Based on knowing the points (or regions) that are covered by the object.
Spatial logic (cont.) • A scene is a represented as a triple (V, Obj, F): • V: N-dimensional space • Mostly N=2 or 3 • Obj: a set of objects in V • F: a mapping from Obj to the powerset of V, i.e. the set of all subsets of V. • Associates each object with a set of points over which it stands, i.e. F(p) := { r V | p stands over r)
Spatial logic (cont.) • V can be partitioned into a grid of rectangular regions. • A scene will be then represented as a discrete scene. • And F will be then a mapping between objects and the regions they cover.
Example e2 I24 I23 p2 I22 I21 p1 e1 I11 I12 I13 I14
Spatial logic (cont.) • Two possible formulations: • Region-based formulation • Object-based formulation
Region-based formulation • Express the positioning of a set of objects with respect to a single region in the scene. • A spatial assertion has the form: where • In addition, there are some shorthand operations: • spatial-eventually (can be +ve or –ve) • spatial-always (can be +ve or –ve)
Semantics • (SE, J , en ) |= p- iff the orthogonal projections on axis en of region g(J) and object p have a nonempty intersection • (SE, J , en ) |= - iff the spatial assertion (SE, J , en ) |= does not hold • (SE, J , en ) |= 12- iff both (SE, J , en ) |= 1 and (SE, J , en ) |= 2
Semantics (Cont.) • (SE, J , en ) |= 1untS+2- iff there esists a region g(J’) which is reached from region g(J) moving along the positive direction of axis en such that assertion 2 holds in region g(J’) and 1 holds in all the regions from g(J) to g(J’) • (SE, J , en ) |= 1untS-2- iff there esists a region g(J’) which is reached from region g(J) moving along the negetive direction of axis en such that assertion 2 holds in region g(J’) and 1 holds in all the regions from g(J) to g(J’)
Shorthands • (SE, J , en ) |= ◊S+/-- will eventually hold in some of the regions encountered moving from region g(J) along axis en • (SE, J , en ) |= S+/-- holds in all the regions encountered moving from region g(J) along axis en
Example em g(J) g(K) p en
Object-based formulation • Express the relationships between objects. • A spatial assertion has the form: ( read: holds in q) where q is an object, expresses that (SE, J , en ) |= holds in any region g(J) containing q • Five different situations occur as follows:
q p p p p (SE, q , x ) |= p - expresses that the projection of q on the x axis is entirely contained in the projection of p
(SE, q , x ) |= ◊S+ p - expresses that every point of the projection of q on the x axis has at least one point of the projection of p to its right side q p p p p p
(SE, q , x ) |= q untS+ p - expresses that the projection of q extends until one point is found that belongs to the projection of p q q p p p p
(SE, q , x ) |= p - expresses that the projection of q on the x axis does not intersect with the projection of p q p p
- expresses that some of the regions of q are aligned with some of the regions of p ((SE, q , x ) |= p ) q q p p p p p p p p p p p
Temporal logic • In general, used to express the ordering of states or actions in time. • A temporal (or state assertion) has the form: where
Semantics • (, k) |= iff the spatial assertion holds in the kth scene of sequence • (, k) |= iff the temporal assertion (, k) |= does not hold • (, k) |= 12iff both (, k) |= 1 and (, k) |= 2 hold • (, k) |= 1untt2iff 2 holds in a scene with index k’ > k and 1 holds in all the scenes form k to k’
Shorthands Two shorthands can be derived • (, k) |= ◊tmeans that will hold in some scene subsequent to the kth one◊t := true untt • (, k) |= tmeans that holds in all the scenesubsequent to the kth onet := ( true untt( ) )
Describing Scene Sequences S1 S2 S3 S4 S5 1 1 1 2 • each node is labeled with the spatial assertion, satisfied in its scene • S1, S2 , S3, S4 and S5 are different states on a time line • (, k) |= ( 1 untt2 ) holds for …………… • (, k) |= ◊t2 holds for ……………… • (, k) |= t ( 1 2 ) holds for ……………..
Sequence Description Using STL • 3D scene-based descriptions are used to avoid ambiguity • Two different descriptions are possible 1) Observer-centered description 2) Object-centered description
Two Different Descriptions • Observer-centered description -- images of the same scene taken from different viewpoints -- valid only when the camera is fixed • Object-centered description -- how one object sees the rest of the scene-- does not depend on the camera position
Example • the spatial positions of the house, h as perceived by the car, c along the course of the scene • considering object-centered descriptions and referring to the reference system Ec associated with the car c • 1 := (SEc , c, x) |= h • 2 := ((SEc , c, x) |= h ) • 3 := (SEc , c, y) |= h • 4 := (SEc , c, z) |= h • 5 := ((SEc , c, z) |= h ) The description of the scene using the above assertions can be as follows:
A System for Image Sequence Retrieval A. Iconic Querying 1) Visual Querying 2) Automatic Parsing B. Retrieval from Database1) Sequence Representation (created manually) 2) Sequence Retrieval
Retrieval from Database • To allow for different level of details in queries, three levels of operations are used. • Level 1: Least details • 3 possibilities (before, overlapping, after) • Level 2: Modest details • 6 possibilities • Level 3: Very fine details • 13 possibilities (STL full representational power)
Result of Retrieval Only two frames of the Sequence are shown
Example 2 (Query Specification) Definition of motion of one car during the play back of the other
Result of Retrieval Two frames of the sequence