Simpsons @ work

2. Simpsons @ work

5. Why temporal segmentation?

8. Terms

9. A Shot • smallest piece of information • elementary building block of a video

10. A Cut • Easy • Most common • 95% of all transitions are cuts

11. A Fade-in • Often used at the start of a chapter • From Black Color to Image • Linear intensity change

12. A Fade-out • Often used at the end of a chapter • From Image to Black Color • Linear intensity change

13. A Dissolve • Used to show that time has passed • Often used as a subchapter • From one Shot to another Shot • Linear intensity change

14. Signals

15. Human vs. Computer • Human: Knows what happens. • Computer: No cognitive understanding. Segmentation has to rely on features or signals.

16. SAD • Sum of absolute differences • Substract each pixel by each pixel • Very easy approach SAD = -

17. amount dark bright RGB-Histogram • Find and count all the different shades of red, green and blue and order them from dark to bright.

18. Histogram Difference = • For red,green and blue: Substract the histograms bin by bin and sum up the difference. - 256 bins

19. No cut No cut No cut CUT No cut No cut No cut 1 Frame Difference Result:

20. cut cut cut cut cut cut cut cut How to find the cuts signal • Find Cuts with a threshold • Works on SAD and Histogram Difference frames

21. cut cut cut cut cut cut cut cut cut Problems with Motion SAD: • SAD and Histogram are very sensitive to camera or object movement. signal frames

22. cut cut cut cut cut cut cut cut cut cut cut cut cut cut cut cut Problems with Flash signal • SAD and Histogram are very sensitive to flash. frames

23. Problems with dissolves & fades • So far we have no features that would indicate a fade-in, fade-out or a dissolve Improvment needed.

25. Sliding Window

26. No cut No cut No cut CUT No cut No cut No cut So far: 1 Frame Difference Result:

27. cut cut cut cut cut cut cut No Cut CUT CUT CUT No cut No cut No cut Now: n - Frame difference Result:

28. Motion & Flash signal signal • Not sensitive to flash anymore • Not sensitive to motion anymore frames frames

29. fade Dissolves & Fades signal • Finally we are able to detect dissolves frames

30. dissolve start dissolve end Problems with long dissolves signal • With this feature we miss the long and slow dissolves. frames

32. Mean and Variance

33. mean blue mean green mean red Mean • Mean is the arithmetic average of the histogram values. • When the histogram changes, the mean moves to left or right.

34. Variance • Variance indicates how much the histogram values are spread around the mean. • When the histogram mean changes, the variance changes, too. vs small big

35. Linear Mean Change in Dissolves signal Result: frames

36. Parabolic Variance Change in Dissolves signal Result: frames

37. Process of Detection • Finds slow dissolves • Finds fade-in / fade-out Search for candidates Approximate mean Approximate variance If small enough: Add disolve Calculate the error made

38. Evaluation • ~ 250.000 frames of video material (ca. 3h)~ 1800 cuts, 90 dissolves / fades • Genres: Sport, Cartoons, News, Music, Commericals, Movies, Documentary • Focus on News ( ABC Lateline ) • 3 Categories: Match, Miss and False Positive

39. Results

40. Demo

41. Questions?