1 / 14

Video Mosaics

Video Mosaics. CS 294/283 Final Project Steve Martin U.C. Berkeley, Fall 2003. Overview. Previous Work Method Overview Metrics for Comparing Video Preliminary Results Future Work Questions?. Previous Work on Mosaics. Image Mosaics

dotty
Download Presentation

Video Mosaics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Video Mosaics CS 294/283 Final Project Steve Martin U.C. Berkeley, Fall 2003

  2. Overview • Previous Work • Method Overview • Metrics for Comparing Video • Preliminary Results • Future Work • Questions?

  3. Previous Work on Mosaics • Image Mosaics • 1970’s: Chuck Close/Salvadore Dali make image mosaics by hand • 1990’s: Papers at SIGGRAPH and commercial programs create image mosaics • Video Mosaics • 2002: Video Mosaic program created at Princeton, work published at NPAR 2002 • Example. . . • 2003: Wells Fargo commercial (done by hand) • Example. . .

  4. Method Overview • Step 1: Get a LOT of source video • Quality of results highly dependant on quality and amount of component clips! • In our case, problem solved! • Got stock video companies donate video to the war effort • We have 200GB of high quality video to play with (more than 3000 clips)! • Step 2: Preprocess video • Create a database of feature values for all video. • Each clip further subdivided into tiles • Size determined by the user. • With 20 by 20 pixel tiles, each movie has 864 tiles. • Metrics are run on each tile, values recorded • This is done with a separate program that crawls the video repository

  5. Method Overview (2) • Step 3: Create video mosaic • Load in original video • Parse the previously generated database • For each clip in the original video • Run the same metrics as was used to generate the database • Search the database for the closest tile • Replace the original tile with one from another movie, continue • Note that we can’t just brute force compare video or we’ll be computing forever. • How to come up with numbers that describe tiles?

  6. 0.5 * + 0.5 * = Video Comparison Metric 1: Average Color • Want replacement clips to have close to or same average color as original tile. • To calculate: • Just add color values of pixels in movie tile and divide. • Actually not terrible by itself. . . • As tiles get smaller, approach pixel size, eye blends them • However, note that average color can be ‘fooled’ • i.e. a clip that is half red and half green averages to olive brown: • Need something else. . .

  7. Vs. Vs. Video Comparison Metric 2: Color Histograms • In addition to average color, want some kind of measure of color frequency. • Helps solve previous problem, although not entirely • To calculate: • For each pixel of each frame of each tile in movie: • Convert from RGB to HSB • Take hue value and histogram • Create histograms on a per-tile basis • Number of histogram buckets is changeable • Usually use around 10.

  8. θ Video Comparison Metric 3: Edge Histograms • Also would like to have some kind of edge comparison between original and replacement tile. • To calculate: • Foreach frame of each tile • Convert frame to black and white • Don’t care about color—want edge data • Resize the image down to 10 x 10 • We want major edges only • Use a bicubic resize to blur the image slightly • For each pixel in the new image, • Calc intensity gradient in the x and y directions • Take the arctangent and histogram. • Do for each frame, create histogram. • Example of this metric at work:

  9. Finding the Best Match • Currently, a linear search through a large unordered flat file. • Parse this into a simple row/column data structure, keep it in memory. • Search not as bad as it sounds; takes about a second with 259200 entries. • Coming soon: some tricks to make this much better. • For each tile of the source movie, compare with each tile entry in the database. • Get the absolute difference between average colors • Get the distance between color histograms • Get the distance between edge histograms • Add difference values to get a value for how good a match this tile is for the original. • Can weight different metric to emphasize different features • Choose entry that minimizes this.

  10. Video vs. Photo Mosaics • Why bother using video? • Could mosaic each frame individually, and then string the frames together. • Example. . . • Would like to exhibit some kind of temporal consistency • I.e. each tile is recognizable as a movie instead of a compression artifact. • Neat effect; less distracting for the eye.

  11. Preliminary Results

  12. Preliminary Results

  13. What’s Next? • Need to implement some way of dividing tile clips into arbitrary length. • Approximation: subdivide tiles into groups of frames. • Index the rest of the video so I can leverage the full library. • Still working on good ways to preserve temporal locality • i.e. place video tiles in a manner that makes sense over time in addition to just per frame • Include metrics that look at neighboring tiles to avoid drastically different edges • These just end up looking like compression artifacts • Princeton’s video has this problem • Different sized tiles • I think a ‘patchwork’ effect would look interesting, albeit harder to compute. • Optimize, optimize, optimize! • Takes forever to make one of these!

  14. References • Klien, Grant, Finkelstein, and Cohen. “Video Mosaics” Proceedings of NPAR 2002. • Microsoft Video For Windows developer’s site • Apple Quicktime SDK developer resources • James O’Brien, Alexander Berg, Charless Fowlkes Questions? Comments?

More Related