A Generic Virtual Content Insertion System Based on Visual Attention Analysis

1. A Generic Virtual Content Insertion SystemBased on Visual Attention Analysis H. Liu1, 2, S. Jiang1, Q. Huang1, 2, C. Xu2, 3 1Institute of Computing Technology, Chinese Academy of Sciences 2China-Singapore Institute of Digital Media 3National Lab of Pattern Recognition, Institute of Automation

2. Outline • Motivation • Related work • The proposed Virtual Content Insertion (VCI) system • Experimental results • Conclusions http://www.jdl.ac.cn

3. Virtual Content Insertion • Convenient • Changeable • Cost less To construct a generic VCI system http://www.jdl.ac.cn

4. Challenge • Advertisement insertion VS. Augmentation • Software based VS. Hardware based • Challenge • Insertion time • Insertion place • Insertion method • Insertion content

5. Related work– Insertion Time • Insert advertisements into video prologue • Be neglected • Insert the ad into interesting segments • Our method • Temporal attention • Higher Attentive Shots • K. Wan, C. Xu, “Automatic Content Placement in Sports Highlights”, ICME, 2006. http://www.jdl.ac.cn

6. Related work– Insertion Place • Static region • Color consistent region • Visual relevance measure • Lower informative region • Our method • Spatial attention • Lower Attentive Region • C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004. • K. Wan, etc., “Automatic Content Placement in Sports Highlights”, ICME, 2006. • Y. Li, etc., “Real Time Advertisement Insertion in Baseball Video Based on Advertisement Effect”, ACM Multimedia, 2005. http://www.jdl.ac.cn

7. Related work – Insertion Method • Challenge • Camera parameters unknown • Existing methods • Structure of the scene • Predefined landmarks • Our method • Affine transformation • Global Motion Estimation • X. Yu, etc., “Inserting 3D Projected Virtual Content into Broadcast Tennis Video”, ACM Multimedia 2006. • C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004. http://www.jdl.ac.cn

8. Related work – Insertion Content • Improve the ad effect • Decrease intrusion • VideoSense • Textual relevance • Local visual and aural relevance T. Mei, X-S. Hua, L. Yang, S. Li, “VideoSense-Towards Effective Online Video Advertising”, 16th ACM International Conference on Multimedia, pp: 1075-1084, 2007 http://www.jdl.ac.cn

9. Outline • Motivation • Related work • The proposed VCI system • Experimental results • Conclusions http://www.jdl.ac.cn

10. http://www.jdl.ac.cn

11. Temporal Attention • Basic idea • The more different a frame/shot/video clip is to the preceding ones, the more probable for it to be attended • Measure • Novelty http://www.jdl.ac.cn

12. http://www.jdl.ac.cn

13. HAS Detection • Shot novelty • The longer a shot is, the more it is probable to be attended http://www.jdl.ac.cn

14. http://www.jdl.ac.cn

15. Spatial Attention Analysis • Static attention • Spatio-temporal attention • Motion saliency • Static novelty http://www.jdl.ac.cn

16. Static Saliency (1) • Psychological basis • Contrast • Information theory • Our method • Contrast and information theory • Calculation • Property of receptive field http://www.jdl.ac.cn

17. Static Saliency (2) • Perceptive unit • Pixel/block • Region • Object • Color quantization http://www.jdl.ac.cn

18. Static Saliency (3) • Contrast • Information density • Saliency http://www.jdl.ac.cn

19. Motion Saliency • Motion Vector Space  HSV color space http://www.jdl.ac.cn

20. Static Novelty (1) http://www.jdl.ac.cn

21. Static Novelty (2) • Static novelty: An event’s importance along temporal axis • Distance: KL http://www.jdl.ac.cn

22. http://www.jdl.ac.cn

23. Static LAR Detection http://www.jdl.ac.cn

24. Dynamic LAR Detection http://www.jdl.ac.cn

25. http://www.jdl.ac.cn

26. Affine transformation http://www.jdl.ac.cn

27. Global Motion Estimation http://www.jdl.ac.cn

28. Outline • Motivation • Related work • The proposed VCI system • Experimental results • Conclusions http://www.jdl.ac.cn

29. Experiment Data Set – Test Video http://www.jdl.ac.cn

30. Experiment Data Set -- Virtual Content http://www.jdl.ac.cn

31. Temporal attention & HAS (1) http://www.jdl.ac.cn

32. Temporal attention & HAS (2) • Noticing rate: • Consistency: the similarity between attention curve and noticing curve http://www.jdl.ac.cn

33. Temporal attention & HAS (3) • Relationship between noticing rate and attention value http://www.jdl.ac.cn

34. Spatial attention & LAR • Invited the users to evaluate the brands he/she has noticed in the video. rate of GOOD http://www.jdl.ac.cn

35. Static Insertion Demo

36. Dynamic Insertion Evaluation • Subjective evaluation • Criteria • Is the result’s deformation consistent with the scene? • Does the inserted VC follow the camera motion? • To what degree the user is satisfied with the result? • Scores: 15 http://www.jdl.ac.cn

37. Dynamic Insertion Result http://www.jdl.ac.cn

38. Conclusion • Main contribution • A generic virtual content insertion system. • A new method of temporal attention and HAS detection • A new method of spatial attention and LAR detection • A dynamic insertion method • Future work • The attention change caused by content insertion • The interaction between insertion time and place http://www.jdl.ac.cn