1 / 17

MINMAX Optimal Video Summarization with Frame Skip Constraint

MINMAX Optimal Video Summarization with Frame Skip Constraint. 1,2 Zhu Li 3 Guido Schuster 1 Aggelos K. Katsaggelos 2 Bhavan Gandhi 1 Department of ECE, Northwestern University, Evanston, USA 2 Motorola Labs, Schaumburg, USA 3 Hochschule fur Technik Rapperswil (HSR), Switzerland. Outline.

olwen
Download Presentation

MINMAX Optimal Video Summarization with Frame Skip Constraint

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MINMAX Optimal Video Summarization with Frame Skip Constraint 1,2Zhu Li 3Guido Schuster 1Aggelos K. Katsaggelos 2Bhavan Gandhi 1Department of ECE, Northwestern University, Evanston, USA 2Motorola Labs, Schaumburg, USA 3Hochschule fur Technik Rapperswil (HSR), Switzerland

  2. Outline Introduction Definitions and Assumptions Formulations Optimal and Heuristic Solutions Solution to the Dual Problem Frame Distortion Metric Simulation Results

  3. Introduction • Why Video Summary • View time constraint, a shorter version is more desirable in some applications, for eg, security, mil and entertainment apps • Storage, Bandwidth and Energy Constraint, a shorter version with better SNR quality conveys more useful information. • Solution: • Video Shot Segmentation, • Key frames selection within a video shot • Previous works: clustering on visual features, curve simplification, utility maximization, etc.

  4. An application scenario 2G/2.5G data channel 6kpbs ~ 15kpbs • Operating at voice rate • Reasonable visual quality • Synchronized with audio if so wish

  5. Video Sequence (n-frame): V= Video Summary (m-frame): Reconstructed Sequence: Summary Distortion: Summary Rate: Definitions and assumptions

  6. Rate-Distortion Formulation • Summarization as a rate-distortion optimization problem • MDOS formulation: • MROS formulation: • Frame skip constrained:

  7. Dynamic Programming Solution • Dynamic Programming • Exhaustive search not practical. • Segment distortion state and rate definitions: MROS summary frame selections: {l0, l1, … lm-1}, l0=0.

  8. The Algorithm • MROS algorithm: • The recursion: • The initial condition:

  9. The DP Trellis • MROS algorithm: n=6, no skip constraint n=6, max skip=3

  10. An Example Solution • Start from R10 • Dmax and Kmax Constrained state transition • Stop when the final virtual frame fn is reached. • Multiple optimal solutions • S*={f0, f4, f7}, { f0, f4, f6} … { f0, f2, f5} f r a m e k epoch t

  11. Frame Distortion Metric • An elusive problem • Application specific • PCA analysis to find the subspace spanned by a large set of video frames • Weighted Euclidean distance in PCA space as frame distortion metric • Works well with subjective perception.

  12. Simulation results : MROS: “foreman” sequence, frames 150~299 n=150, Dmax = 6.4, Kmax= no constraint Results: m=25

  13. Simulation results: skip constrained: MROS: “foreman” sequence, frames 150~299 n=150, Dmax = 6.4, Kmax= 10 Results: m=32.

  14. A Heuristic Solution • Distortion Constrained Skip (DCS) algorithm: • Set L=0, add fL to the summary S • FOR k=1 TO n • IF d(fL, fk) > Dmax • L=k, • add fL to the summary S • END • END DCS is the optimal solution if:

  15. Rmax D* Solution to the MDOS formulation Bi-Section searching on the operational R-D function: • The ORD is non-increasing (lemma 1.) • Bi-section search on the distortion, and solve for each distortion with MROS solution. • Will converge to the optimal solution D*.

  16. Conclusion and Future Work • The solution is rate-distortion optimal • The heuristic DCS algorithm is near optimal most of time and quite efficient • Summaries operates at voice rate suitable for 2G and 2.5G deployment (demo): • “bond” sequence at 13.2kpbs, Dmax=6.0 • “bond” sequence at 11.7kpbs, Dmax=8.0 • “bond” sequence at 9.7kpbs, Dmax=12.0 • “foreman” sequence at 10.8kpbs, Dmax= 6.0 • “foreman” sequence at 9.4kpbs, Dmax=8.0 • “foreman” sequence at 8.4kpbs, Dmax=12.0 • Future work: bit constrained MINMAX summarization.

  17. Questions ? ? ……. brigado !

More Related