1 / 27

Video Summarization using Determinantal Point Processes (DPP)

This study explores the use of DPPs for video summarization, focusing on sequential DPPs and large-margin training methods. It delves into the challenges of representativeness and diversity in frame selection, proposing structured prediction and submodular functions for subset modeling. Experimentation with various datasets and features showcases the effectiveness of Sequential DPP in video summarization. Further advancements include learning parameters via max likelihood estimation and the potential for incorporating neural networks. Future directions aim to enhance inference algorithms and explore models beyond DPP.

patriciaperez
Download Presentation

Video Summarization using Determinantal Point Processes (DPP)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Video Summarization viaDeterminantal Point Processes (DPP) Boqing Gong University of Southern California Joint work with Wei-Lun Chao, Kristen Grauman, and Fei Sha

  2. Background • Basic idea of DPP • Sequential DPP (NIPS 2014) • Large-margin training of DPP • Conclusion

  3. Background • Motivation: Indispensable for fast video browsing and retrieval • Representation: • Key frames / segments extraction • Subset Selection problem

  4. Background • Video summarization is hard: • Individual selected frame: Representativeness • Selected frames as a whole: Diversity • Naïve solution:Clustering • Competing !

  5. Background • Clustering works?

  6. Video summarization: an overview • Video summarization is hard: • What criteria lead to user perspective? • What kind of models: • Supervised learning ! • Diverse subset with representative items

  7. Background • How to model subset selection problem? • Structured prediction, submodular functions • Determinantal Point Processes (DPPs) [Alex Kulesza and Ben Taskar, 2012]

  8. Background • Basic idea of DPP • Sequential DPP (NIPS 2014) • Large-margin training of DPP • Conclusion

  9. Basic idea of DPP • Idea: A point process based on matrix determinant. • Formulation:M discrete items (binary decision)

  10. Basic idea of DPP • Why diverse? • Extreme cases:

  11. Basic idea of DPP • Learning in DPP: • 11

  12. Background • Basic idea of DPP • Sequential DPP (NIPS 2014) • Large-margin training of DPP • Conclusion

  13. Sequential DPP • Motivation: • The temporal structure of video is missing • Proposed Idea: • Sequential DPP via Markov properties

  14. … • …

  15. Sequential DPP • Modeling the sequential structure: • Conditional DPP: still a DPP !

  16. Sequential DPP • Parameterization:

  17. Inference and Learning • Inference: • Allow brute-force search in small chunks • Optimization:

  18. Sequential DPP • Experimental setting: • 3 datasets: OVP (50), Youtube (39), Kodak (18) • Fisher vectors + Saliency + Contextual features • Evaluation: Recall, Precision, and F1 score • Comparison: unsupervised methods & vanilla DPP

  19. Sequential DPP • Experimental Results:

  20. Sequential DPP • Experimental Results:

  21. Sequential DPP • Experimental Results:

  22. Background • Basic idea of DPP • Sequential DPP (NIPS 2014) • Large-margin training of DPP • Conclusion

  23. Learning parameters in DPP • Maximum likehood estimation • Focuses on observed data only • Large-margin training • Maximizes margin between observed and undesired data • Discriminative learning • More flexible: incorporating evaluation metrics

  24. Large-margin training of DPP • More discriminative and flexible

  25. Conclusion • Supervised learning for video summarization • DPPs: modeling diversity subset selection • Video structure: Sequential DPP • Parameterization: Neural networks • Future work • Better inference algorithms • Models beyond DPP (submodular)

More Related