1 / 22

Clustering video scenes

Clustering video scenes. Nebojsa Jojic. Six break points vs. six things in video. Traditional video segmentation: Find breakpoints Example: MovieMaker (cut and paste) Our goal: Find possibly recurring scenes or objects. timeline. 1. 2. 3. 2. 4. 1. 4. 3. 2. 3. 2. 3. 5. 6.

yitro
Download Presentation

Clustering video scenes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Clustering video scenes Nebojsa Jojic

  2. Six break points vs. six things in video • Traditional video segmentation: Find breakpoints Example: MovieMaker (cut and paste) • Our goal: Find possibly recurring scenes or objects timeline 1 2 3 2 4 1 4 3 2 3 2 3 5 6

  3. Six break points vs. six things in video • Differences: timeline • A class is detected at multiple intervals on the timeline. For example, class 1 models a baby’s face. Break pointers miss it at the second occurrence. The class occurs more in the rest of the sequence 1 2 3 2 4 1 4 3 2 3 2 3 5 6

  4. Six break points vs. six things in video • Differences: timeline One long shot contains a pan of the camera back and forth among three scenes (classes 2,3 and 5) 1 2 3 2 4 1 4 3 2 3 2 3 5 6

  5. Six break points vs. six things in video • Traditional video segmentation: Find breakpoints Example: MovieMaker (cut and paste) timeline Two shots detected just because the camera was turned off and then on with a slightly different vantage point are considered a single scene class. 1 2 3 2 4 1 4 3 2 3 2 3 5 6

  6. Model • Appearance mean variance • Camera/object motion • Temporal constraints • Unsupervised learning – the only input is the video

  7. TMG: Fitting a generative model Class index Class mean (representative image) Shift Mean with added variability Transformed (shifted image) Transformed image with added non-uniform noise

  8. Mean • One class summary • Variance 5 classes Example • Hand-held camera • Moving subject • Cluttered background DATA

  9. Current implementation • DShow filter for frame clustering (5-15 frames/sec!) • Translation invariance • On-line learning algorithm • Classes repeating across video • Potential applications: • Video segmentation • Content based search/retrieval • Short video summary creation • DVD chapter creation

  10. Example: Clustering a 20-minute whale watching sequence

  11. Learned scene classes

  12. A random interesting 20s video

  13. 0 min 9 min

  14. Grouping based on class transition patterns class … time

  15. Automatic DVD chapter creation …

  16. Cluster means for panorama TMG

More Related