1 / 35

Image Fusion for Context Enhancement and Video Surrealism

Image Fusion for Context Enhancement and Video Surrealism. Ramesh Raskar Mitsubishi Electric Research Labs, (MERL). Adrian Ilie UNC Chapel Hill. Jingyi Yu MIT. Dark Bldgs. Reflections on bldgs. Unknown shapes. ‘Well-lit’ Bldgs. Reflections in bldgs windows. Tree, Street shapes.

hallie
Download Presentation

Image Fusion for Context Enhancement and Video Surrealism

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Fusion for Context Enhancement and Video Surrealism Ramesh Raskar Mitsubishi Electric Research Labs, (MERL) Adrian Ilie UNC Chapel Hill Jingyi Yu MIT

  2. Dark Bldgs Reflections on bldgs Unknown shapes

  3. ‘Well-lit’ Bldgs Reflections in bldgs windows Tree, Street shapes

  4. Night Image Background is captured from day-time scene using the same fixed camera Context Enhanced Image Day Image

  5. Mask is automatically computed from scene contrast

  6. But, Simple Pixel Blending Creates Ugly Artifacts

  7. Pixel Blending

  8. Pixel Blending Our Method:Integration of blended Gradients

  9. Outline • Context Enhancement • Gradient-based Fusion • Video Enhancement • Surrealism

  10. Gradient field Nighttime image x Y I1 G1 G1 Mixed gradient field x Y G G Importance image W I2 x Y G2 G2 Final result Daytime image Gradient field

  11. Reconstruction from Gradient Field • Problem: minimize error |Ñ I’ – G| • Estimate I’ so that G = Ñ I’ • Poisson equation • Ñ 2 I’ = div G • Full multigrid solver GX I’ GY

  12. Why Gradient-based Approach • Comparison of intensity values are important • Maintain gradients to capture local variations • Directly solve for desired gradients • Maintain subtle details • Mix dissimilar images • No need for precise segmentation

  13. Comparison • Average • Subtle details are lost • Pixel-wise blending • Sharp transitions

  14. Issues • Boundary conditions • Color shifts

  15. Boundary Conditions • Assumed Neumann condition at borders, • Ñ I’ · N = 0, • Enforced by haloing image with blacks

  16. Color Shift • Mixing dissimilar images • Goal: final image appearance matches input images at corresponding pixels Ifinal(x,y) = c1Ipoisson(x,y) + c2 • Solve Wi(x,y) Ioriginal(x,y) = c1Ipoisson(x,y) + c2 • Each color channel reconstructed separately

  17. Outline • Context Enhancement • Gradient-based Fusion • Video Enhancement • Surrealism

  18. Overview of Process Original night time traffic camera 320x240 video Day time image: By averaging 5 seconds of day video Input Output Enhanced video Note: exit ramp, lane dividers, buildings not visible in original night video, but clearly seen here. Mask frame (for frame above): Encodes pixel with intensity change

  19. Algorithm Frame N Gradient field Mixed gradient field TimeAveraged importance mask Processed binary mask Final result Gradient field Frame N-1 Daytime image

  20. Outline • Context Enhancement • Gradient-based Fusion • Video Enhancement • Related Work • Surrealism

  21. Related Work • Spatio-temporal Composition • Duchamp (Nude descending a staircase) • Freeman 2002 • Fels 1999, Klein 2002, Cohen 2003 • Gradient-based Techniques • Multi-spectral: Socolinsky 1999 • Shadow removal: Weiss 2001 • High dynamic range: Fattal 2002 • Image editing: Perez 2003 • Some at Siggraph’04

  22. Surrealism Rene Magritte, ‘Empire of the Light’

  23. Outline • Context Enhancement • Gradient-based Fusion • Video Enhancement • Surrealism

  24. Time-lapse Mosaics Maggrite Stripes time

  25. Time Lapse Mosaic

  26. Time Lapse Mosaic

  27. t

  28. Sunrise at Night

  29. BiSolar System

  30. Discussion • User Experience • More effective in conveying scene context • ‘Dreamy’ appearance • Nonrealistic : False conditions • Applications • Tools for artists • Surveillance • Amusement park rides • Performance • ~1 sec/frame for 320x240 • ~ 3 min for 4Mpixel image

  31. Image Fusion for Context Enhancement • Nonrealistic but comprehensible context • Fusion using multiple images • Enhancing night images with day bgrnd • Gradient-based fusion • Video surrealism tools t

More Related