Webcam-synopsis: Peeking Around the World

1. Webcam-synopsis:Peeking Around the World Young KiBaik (CV Lab.) 2008. 4. 4 (Fri)

2. Webcam-Synopsis: Peeking Around the World • References • Webcam Synopsis : Peeking Around the World • Yael Pritch, Alex Rav-Acha, Avital Gutman Shmuel Peleg • (ICCV 2007) • Non-Chronological Video Synopsis and Indexing • Yael Pritch, Alex Rav-Acha, Shmuel Peleg (PAMI 2008) • Background Cut • J. Sun, W. Zhang, X. Tang, and H. Shum (ECCV 2006)

3. Webcam-Synopsis: Peeking Around the World • What is Video-Synopsis? Video synopsis is compact video data reducing spatio-temporal redundancy in input video.

4. Webcam-Synopsis: Peeking Around the World • Demo (Final result)

5. Webcam-Synopsis: Peeking Around the World • How can we make Video Synopsis? • Basic concept • Detect activity tubes… in input video I. • Put activity tubes… to the S… considering relation between - tubes and background. - tube and other tubes.

6. Webcam-Synopsis: Peeking Around the World • Synopsis • Properties of video synopsis S • The video synopsis S should be substantially shorter than the original video I. • Maximum “activity” from the original video should appear in the synopsis video. • The dynamics of the objects should be preserved in the synopsis video. (for example, regular fast-forward may fail to preserve the dynamics of fast objects.) • Visible seams and fragmented objects should be avoided.

7. Webcam-Synopsis: Peeking Around the World OBJECT-BASED SYNOPSIS

8. Webcam-Synopsis: Peeking Around the World • Object-based synopsis construction • Online Phase • This phase is done in real time during video capture. • Preparing step for response phase. • Response Phase • started with user query. • This phase may take a few minutes, depending on the activity in the time period of interest.

9. Webcam-Synopsis: Peeking Around the World • Overall flow

10. Webcam-Synopsis: Peeking Around the World ONLINE PHASE

11. Webcam-Synopsis: Peeking Around the World • Video Acquisition • Video Volume Data t y I(x, y, t) : 3D Space Time Volume (x, y) : Spatial coordinates of the image t : Frame index x

12. Webcam-Synopsis: Peeking Around the World • Background Video Construction • Temporal Median t Temporal Window y Sorting by intensity Selecting median value x 4 minutes

13. Webcam-Synopsis: Peeking Around the World • Background Video Construction t t y y x x I B

14. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Using Background Cut (ECCV 2006) • Using min-cut to get a smooth segmentation of foreground objects. • Only considering 2D image (not video).

15. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Extracting moving objects • We have information for… I(x, y, t) : 3D Space Time Volume B(x, y, t) : 3D Space Time Volume for background Current ~ : Current image = I (x, y, current time) : Current background image = B (x, y, current time)

16. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Simplify Object detection problem to… • Binary labeling f: Foreground = 1, Background = 0 • Obtaining the desirable labeling f … → with Gibbs energy function. Color term Contrast term

17. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Extracting moving objects • Unary term (or color term) Color differences between image and background Foreground energy Background energy

18. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Using Background Cut (ECCV 2006) • Using min-cut to get a smooth segmentation of foreground objects. • Only considering 2D image (not video).

19. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Extracting moving objects • Binary term (or contrast term) In case of same labeling, energy is decreased. In case of different labeling, energy defends on differences between neighboring intensities.

20. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Using min-cut algorithm… - Moving objects are extracted.

21. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Extracting moving objects • Constructing a mask of all foreground pixels in space-time volume. • Applying a 3D morphological dilation on mask. • Finally we can obtain activity tubes (or objects). t x Example of tubes

22. Webcam-Synopsis: Peeking Around the World • Moving Objects(tubes) Extraction • Object queue • Obtained activity tubes are saved in queue. End of online phase…

23. Webcam-Synopsis: Peeking Around the World RESPONSE PHASE

24. Webcam-Synopsis: Peeking Around the World • Response phase • User query • “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.” • “I would like to watch in five minutes a synopsis of the last week.”… When user query occurred, response phase are started. Input video Synopsis

25. Webcam-Synopsis: Peeking Around the World • Response phase • User query • “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.” • “I would like to watch in five minutes a synopsis of the last week.”…

26. Webcam-Synopsis: Peeking Around the World • Creating time lapse background video • Time lapse background video (Bout) • The background of the synopsis video • It should represent the background changes over time. • Day-night transitions, etc. • It should represent the background of the activity tubes. Time lapse BG video Background Video Sampling uniformly

27. Webcam-Synopsis: Peeking Around the World • Creating time lapse background video • Time lapse background video • A temporal activity histogram Ha • A uniform temporal histogram Ht

28. Webcam-Synopsis: Peeking Around the World • Creating time lapse background video • Time lapse background video • A temporal activity histogram Ha • A uniform temporal histogram Ht • Interpolating the two histograms Hi = Ha + (1- λ)Ht Time lapse BG video Background Video (b) Sampling with Hi Bout

29. Webcam-Synopsis: Peeking Around the World • Selecting tubes and stitching • Definition : temporal mapping (or time shift) : tube with time segment : shifted tube with time segment Input video Synopsis

30. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • Energy for temporal mapping M : Queue : target tube : other tube : user selected weights

31. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • Graph Node number = number of tube Longest tube tube tube Synopsis tube t tube label label Label = number of frame t label label

32. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • Energy for temporal mapping M • Unary term • Activity Cost Synopsis Only pixel that were not entered into the synopsis are added to the activity cost.

33. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • Energy for temporal mapping M • Unary term • Consistency with background Synopsis : border of the mapped activity tube

34. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • Energy for temporal mapping M • Binary term • Collision Cost Synopsis This expression give a low penalty to pixel whose color is similar to the background…

35. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • Energy for temporal mapping M • Binary term • Temporal Consistency Cost Synopsis Preserving the chronological order of events

36. Webcam-Synopsis: Peeking Around the World • Select tubes and stitching • E(M) is minimized by Min-cut algorithm. • Stitching the synopsis video • To make more reliable result… → The α–Poisson Image Blending End of response phase…

37. Webcam-Synopsis: Peeking Around the World • Conclusion • The method to creating a short video that is a synopsis of and video stream has been presented. • Discussion • Contribution • Found out new interesting application. • Can we find more contribution? • Finding higher dimensional problem and Solving it. • 3D volume animation, etc.