1 / 40

Video Processing

Video Processing. Video is a rich information source frames (individual images) links between frames (cuts, fades, dissolves) changes in color, shapes, motion of both camera and objects acquisition (shot angles, camera motion)

monte
Download Presentation

Video Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Video Processing • Video is a rich information source • frames (individual images) • links between frames (cuts, fades, dissolves) • changes in color, shapes, motion of both camera and objects • acquisition (shot angles, camera motion) • each type of video has its own characteristics (commercials, news, sports) Video Processing

  2. Video Structure • Frame: typically 1/25 or 1/30 seconds • Shot: sequence of similar frames • elementary video units • a single event • Clip / Scene: sequence of shots consecutive in time, space, action • Episode: consecutive scenes • intro, news, reporter, weather Video Processing

  3. Video Structure (cont.d) Del Bimbo 99 Video Processing

  4. Video Retrieval • A video can be accessed at the • structural level: browsing, retrieval of shots, scenes, episodes • content level: according to camera motion, motion of characters or objects, audio properties, scenes, semantics of color, texture, shape, object properties … Video Processing

  5. Video Partitioning • Shot extraction and classification of editing effects due to • camera breaks (cuts): abrupt transitions • gradual transitions: dissolves, wipes, fade-in/out • camera movements: panning, tilting, zoom • Simple methods for camera breaks • More sophisticated methods for gradual transitions and camera movement Video Processing

  6. Cut: Frames Between Shots shot1 Furht. et.al 96 shot2 Video Processing

  7. Dissolve: Transition Frames Furht. et.al 96 Video Processing

  8. Uncompressed Video Partitioning • Detect boundaries of consecutive camera shots • compare adjacent frames • for camera breaks compare color histograms of adjacent frames • for gradual transitions and camera motion histograms are less successful • Other techniques are based on edge detection, motion analysis etc. Video Processing

  9. Histograms • Grey level histograms for 3 successive frames • Frames 1 and 2 almost identical • Camera break between 2 and 3 • Compute histogram differences Video Processing

  10. Color Histograms • H(I,v) : number of pixels in I with intensity v • MxN pixels • Grey-level images: 8 bits/pixel - 256 bins in histogram • Color images: 24 bits/pixel - 224 bins • Convert color to YUV color space and process intensity only: • I = 0.299R + 0.587G + 0.114B Video Processing

  11. 1. Camera Breaks • Pair-wise pixel comparison (intensities) • Histogram comparison for camera breaks • threshold selection • twin comparison approach • multi-pass approach • Motion vector analysis for camera motion and gradual transitions • Hough transform • Video X-ray Video Processing

  12. Pair-Wise Pixel Comparison • Count pixels changed from a frame to the next • A shot boundary is found if more than Tbpixels changed • Problem: sensitivity to camera/object motion and noise • many pixels change Video Processing

  13. Pair-Wise Block Comparison • Compare blocks instead of pixels • μi,μi+1: mean intensity values in frames • si,si+1: variances • Less sensitive to motion and noise • t: does not change for different video sources Video Processing

  14. Pair-Wise Histogram Comparison Furht. et.al 96 • Even less sensitive to motion • i: frame count • j: intensity count in H • G=MxN intensities Video Processing

  15. Histogram Comparison gradual transitions camera breaks Furht et.al. 96 Video Processing

  16. Thresholds • Tolerate variations while ensuring good performance • low thresholds accept many false positives • high thresholds reject true transitions • Threshold: varies from one video source to another • e.g., cartoons exhibit larger frame differences than films Video Processing

  17. 2. Gradual Transitions • Transitions not as high as in camera breaks • dissolve, fade-in/out, other special effects • high transitions in a neighborhood • lower thresholds do not solve the problem Furht. et.al 96 Video Processing

  18. Twin-Comparison • Two thresholds • Tb for camera break detection • Ts < Tb for special effects like dissolves, motion • Compare consecutive frames (e.g. histograms) • if difference exceeds Tb: camera break • if difference exceeds Ts: potential cut • Accumulate differences from that frame until the transition becomes lower than Ts • A boundary is detected if the accumulated difference becomes higher than Tb Video Processing

  19. Twin Comparison Example camera break special effect Furht. et.al 96 shot boundary Video Processing

  20. Threshold Selection • Based on the distribution of the frame-to-frame histogram changes • Most changes are due to noise ~90%, scene changes ~10% • Scan entire video and compute distribution of changes (e.g., histogram differences) • Assume Gaussian distribution and compute: μ, σ • Compute the two thresholds as • Tb=μ+ασ,α=4-6 • Ts=βμ, β=1.5-2 Video Processing

  21. Motion Vector Analysis (object motion) • Detect camera breaks using motion vectors (MV) • Compute MVs by block matching • Compute correlation of the same block bi from frame i to frame i+1 • Assign a displacement vector Dbi to bi • The same for all blocks between frames Video Processing

  22. Motion Smoothness • For each frame compute Wi • Nominator counts significant motion vectors in frame i • Denominator counts significant transitions in motion vectors • Wi0 indicates camera break Video Processing

  23. 3. Camera Motion • Detect changes due to camera movement • Camera zoom in/out, tilting/panning • Transitions resemble gradual transitions • More specific techniques • Analysis of motion vectors Furht. et.al 96 Video Processing

  24. Camera Panning • Most motion vector exhibit same direction • Σb|θb - θm| < Θp • Θb is the direction of the vector of block b • θm is the direction of the entire set of blocks • The variation Θp should be close to 0 Furht. et.al 96 Video Processing

  25. Camera Zooming • Assume focus center within the frame and little object motion at the periphery of a frame • Compare v in top/bottom rows, left/right columns • Every column |vtop-vbottom| >= max(|vtop|,|vbottom|) • Every raw |vleft-vright| >= max(|vleft|,|vright|) • Zooming: most vectors satisfy these condition Video Processing

  26. Compressed Video Partitioning • Frame comparison using • DCT coefficients instead of blocks • MPEG motion vectors • Combination of the above • Same techniques using I frames only (faster) Furht. et.al 96 Video Processing

  27. Pair-Wise DCT Comparison • Applies to the DCT coefficients of corresponding blocks in consecutive I frames which are f-distance apart • k=1...64 coefficients • For each block computes Video Processing

  28. Threshold Selection • Same techniques for threshold selection with uncompressed video • Diffl > t : a block has changed • t: does not vary with video sources • Shot boundary: the percentage of blocks that changed exceeds Tb • Tb varies for different video sources Video Processing

  29. Example D values between successive frames Furht. et.al 96 • Sharp peaks indicate camera breaks • Cannot handle gradual transitions, camera or object motion • Twin comparison using Ts, Tb thresholds • Applies only to I frames (no DCTs for P, B frames) • Faster but, many false positives Video Processing

  30. Motion Vectors • Motion vectors are associated with P, B frames • The residual error between blocks is DCT encoded • If the error is large DCT encode the original blocks and no motion vectors are stored in this case • Many blocks with no motion vectors indicatecamera break • Camera break: motion vectors M <Tb ~ 0 Video Processing

  31. Motion Vectors (cont.d) • Many false positives for static frames (frames with no motion) • Camera breaks: deep and narrow gaps in diagrams with number of vectors • Combine with D(i,i+f): camera break when high D(i,i+f) with M < Tb but large gap means no motion (static frames) and not transition • Difficult to detect gradual transitions Video Processing

  32. Example Furht. et.al 96 shot boundary M < Tb static frames Video Processing

  33. Hybrid Multiple Pass Approach • First step: DCT comparison on I frames to locate regions of potential interest • low spatial resolution (large f) : very fast • advantage: gradual transitions are likely to be detected because of the large skip factor • disadvantage: many false positives • Second step combining DCTs and motion vectors • smaller skip factor • at the vicinity of candidate boundaries • High processing speed in achieved Video Processing

  34. Camera Motion - Object Motion • Detect specific patterns of motion vectors • Similar techniques with uncompressed video • Motion vectors are provided by MPEG P, B frames • Similar results Furht. et.al 96 Video Processing

  35. Video Browsing • Select a key-frame from each shot • First, middle, last, average frame of shots, I frames for compressed video … • Image retrieval based on key-frames key-frames A.Smeaton,DCU Video Processing

  36. Hierarchical Browsing • Problem: large number of key-frames • Solution: organize key-frames hierarchically • video at the top, key-frames for scenes, shots are lower hierarchical video browser A.Smeaton,DCU Video Processing

  37. Hierarchical Browser Furht et.al. 96 Video Processing

  38. Comments on Video Segmentation • Histograms are sufficient in most cases • Audio could help (silence between shots) • Only one pass through the entire video • Computational cost and delay can be high • A pass at reduced spatial resolution detects potential changes (comparisons every k frames) • Processing at the vicinity of changes to verify the results Video Processing

  39. Further Reading • Centre for Digital Video Processing, Dublin University http://www.cdvp.dcu.ie • B. Furht et.al. “Video and Image Processing in Multimedia Systems”, Chapter 12-14, Kluwer, 1996 • H.J.Zhang, A. Kankanhalli, S.W.Smoliar, “Automatic Partitioning of Full-Motion Video”, Multimedia Systems, 1(1):10-28, 1993 • V. Kobla, D. Doermann, C. Faloutsos “VideoTrails: Representing and Visualizing Structure in Video Sequences” ACM Multimedia 97, Nov. 1997 • O. Marques, B. Furhrt, “Content Based Image and Video Retrieval”, Kluwer Academic Publishers, 2002 • V. Kobla, D. S. Doermann, K-Ip (David) Lin, C. Faloutsos “Compressed domain video indexing techniques using DCT and motion vector information in MPEG video” Proc. of the SPIE Conf. on Storage and Retrieval for Image and Video Databases, Vol. 3022, Feb. 1997 Video Processing

  40. References • S. Lefevre, J. Holler, N. Vincent, “A review of Real-Time Segmentation of Uncompressed Video Sequences for Content-Based Search and Retrieval”, RFAI publication: Real Time Imaging • C. Doulaverakis, V. Vagionitis, M. Zervakis and E. Petrakis: "Adaptive Methods for Motion Characterization and Segmentation of MPEG Compressed Frame Sequence", Intern. Conference on Image Analysis and Recognition (ICIAR'2004), Porto, Portugal, Sept./Oct. 2004, Proc. Part I, Springer Verlag (LNCS 3211), pp. 310-317. Video Processing

More Related