VOGUE

1. VOGUE The Video Object Generator with User Environment Ecole Nationale Supérieure des Mines de Paris, France Instituto Superior Técnico, Portugal Universitat Politècnica de Catalunya , Spain University of Hannover, Germany

2. Objective • To provide an interactive authoring tool to create video objects suitable for MPEG-4 encoding. • Characteristics: • Ease of use • No expert knowledge is needed • High performance

3. Algorithms included • Spatial segmentation segmentation of still images • Temporal segmentation detection of moving objects • Tracking following the object through the video sequence The three algorithms are efficiently combined to provide the user with a performant tool

4. Correction of the results Initial mask Spatial segmentation Tracking Fine segmentation Correction of the results Initial mask Temporal segmentation Initial mask

5. Graphical User Interface (GUI) • GUI provides interface to automatic segmentation + user interaction • GUI supports: • interaction between the different algorithms • general as well as algorithm specific user interaction • User interaction should be minimal but effective

6. The Graphical User Interface

7. The spatial segmentation • It is based on a family of nested partitions of the input image. • A certain level of the family is reached through the fusion of two or more regions of a finer level. • The finest level of the family corresponds to the fine mosaic resulting from flooding a gradient image from all minima. The coarsest level corresponds to the partition that contains the whole image.

8. Object Original image Partition family Region selection The spatial segmentation

9. The spatial segmentation • The whole family of partitions is created during a single morphological flooding of the input gradient image very fast calculation. • The hierarchy of fusions is stored in the form of a tree (typically less than 1000 nodes for QCIF images) and most of the calculations issued from the interaction with the user are carried out on this tree the results of the interaction are perceived as immediate by the user.

10. Object Tracking: Approach Projection of a homogeneous color partition.

11. Object Tracking: Partition Projection • Estimate the motion between the previous and current frames: • Link motion vectors. • Motion compensate the previous texture partition: • Backward block matching. • Fitting of the projected partition into a fine partition of the current image: • Two steps.

12. Object Tracking: Partition Projection • In the fitting process, memory of the original label of the objects is kept (e.g.: the hat). • Small uncertainty areas usually remain. First step: Geometry Second step: Geo. and color

13. Temporal Segmentation

14. Conclusions • A coherent approach has been chosen that combines morphological multiscale representations with motion analysis. • This allows for processing times which are compatible with user interaction on an ordinary PC. • The integration into a common user environment has lead us to analyze the degree of freedom to be left to the user.

15. Future • VOGUE offers a solid ground to be developed in two directions: • As part of a professional MPEG-4 authoring tool • As an editing tool targeting the emerging mass market of numerical photography and video