Personalized Abstraction of Broadcasted American Football Video by Highlight Selection

1. Personalized Abstraction of Broadcasted American Football Video by Highlight Selection Noboru Babaguchi (Professor at Osaka Univ.) Yoshihiko Kawai and Takehiro Ogura (NHK) Tadahiro Kitahashi (Professor at Kwansei Gakuin Univ.) IEEE Transactions on Multimedia, 2004

2. Outline • Introduction • Related Work • Method of detecting significant events in video stream • Method of generating video abstracts • Experimental results • Conclusion

3. Introduction • Video abstract • Creating shorter video clips or video posters from an original video stream • Two schemes of video abstraction • Temporally compressing the amount of the video data • Smith et al., Lienhart et al., He et al., Oh et al., Babaguchi • Provide image keyframe layouts representing the whole video contents • Yeung and Yeo, Uchihashi et al., Chang et al., Toklu et al.

4. Introduction • This method of abstracting sports video • Specifically broadcasted TV programs of American football • Take personalization into consideration • Belong to first scheme of video abstraction • Abstraction based on highlights that are closely related to semantical video contents • Detecting significant events like score events

5. Introduction • How to detect events • Image analysis is very difficult • This method’s solution is to make use of external metadata, called gamestats • Linking video segments with descriptions of the gamestats • Personalization • Extensively attempted in a variety of application fields • Emphasize it because the significance of scenes vary according to preferences and interests • Provide a profile to collect personal preferences

6. Related work – time compression • Smith et al. • Extracted significant information from video such as keywords, specific objects, camera motions and scene breaks with integrating language, audio, and image analyzes • Lienhart et al. • Assemble and edit scenes of significant events in action movies, focusing the on actor/actress’s closeup, text, and sound of gunfire and explosion • These two method are based on surface features of the video rather than on its semantical contents.

7. Related work – time compression • Oh et al. • Abstracting video using user selected interesting scenes • Automatically uncover the remaining interesting scenes in the video by choosing some interesting scenes • Babaguchi • Video abstraction based on its semantical content in the sports domain • To select highlights of a game, an impact factor for a significant event in two-team sports was proposed • He et al. • Create summaries for online audio-video presentations • Use pitch and pause in audio signals, slide transition points in the presentation, and users’ access patterns

8. Related work – spatial expansion • Goal • Visualize the whole contents of the video • Yeung and Yeo • Automatically create a set of video posters (keyframe layouts) by the dominance value of each shot • Uchihashi et al. • Making video posters whose size can be changed according to the importance measure • Chang et al. • Make shot-level summaries of time ordered shot sequence or hierarchical keyframe clusters, as well as program-level summaries

9. Detection of significant events • Detect significant events in the original video stream according to the description in the gamestats

10. Identification of event frames • An event occurs in the shot including the event frame • Attempt to recognize text expressing the game time in the overlay, and then to identify an event frame • To identify the event frame, an overlay model is employed

11. Detection of event shots • A shot is defined as consecutive image frames at a single camera view • Classify the event shots into four types • live-play, replay, pre-play, and post-play shot

12. Generation of personalized video abstract • Generating video abstracts from the detected significant events • Select highlights of the game from all the events, considering profile descriptions • The generating rules for the video abstract:

13. Profile • A video abstract has to be personalized because significance of events could change individually • Provide a profile to collect personal preferences and interests, the items are: • Favorite teams • Favorite players • Events to want to see • Specifications • Range of the video stream to be abstracted • Length of the abstract

14. Significance degree of events • The highlights of the game depend on the significance of each event, and significance can be estimated in terms of event rank, event occurrence time, and the profile • Event rank • State change event (SCE): a score event can change the current state into a different state • Rank 1: SCE’s. • Rank 2: not SCE’s, but exceptional score events • Rank 3: events closely related to score events • Rank 4: all other event that are not Rank 1 to 3 • Rank based significance degree of event Ir: where ri denotes the rank of the ith event Ei, αis a coefficient to consider how large the difference of the rank affects the significance

15. Significance degree of events • Event occurrence time • The score events occurring at the latter or final stage of the game largely affect the result that should have great significance • Occurrence time based significance degree of event It: where N is the number of all events, β is a coefficient to consider how large the occurrence time affects the significance • Profile • Comparing the descriptions of the profile and the occurring event • Profile based significance degree of event Ip: where l denotes the number of descriptions that don’t coincide with each other, γ is a coefficient to consider how large the profile affects the significance • Significance degree of an event I:

16. Selection of highlights • To determine highlights, we concentrate on both priority order of shots and significance degree of events • Priority order of each shot segment • Motion live shot • Still live shot • Motion replay shot • Still replay shot • Motion pre-play shot • Still pre-play shot • Motion post-play shot • Still post-play shot

17. Selection of highlights

18. Experimental results

19. Experimental results

20. Experimental results • Two measures to evaluate the quality of the generated abstract where N denote the number of highlights included in abstracts

21. Experimental results – effect of personalization • Inclusion ratio: the ratio of the length of shots which are concerned with the specified team to the total length

22. Experimental results – effect of personalization 4-symbol string in the cells of table represents each condition of the pre-play, live, replay, and post-play shots for the event

23. Experimental results – effect of personalization

24. Experimental results – effect of personalization

25. Conclusion • Based on the detected significant events by recognizing the textual overlays • Link the video contents with useful external metadata by using the gamestats • Three sorts of significance degrees play a central role in highlight selection • Remaining problems • The method is for different two-team sports • A tailoring mechanism for shots, adjusting for the total abstract • Seek for a sophisticated way of refining the profile