1 / 77

Change Detection in Rolling Shutter Cameras

This seminar talk discusses change detection in rolling shutter cameras. It covers the challenges in detecting changes, the effects of rolling shutter motion blur, and the methods used to estimate camera motion and detect changes. The talk also compares the results with global motion deblurring techniques.

bsutton
Download Presentation

Change Detection in Rolling Shutter Cameras

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Change Detection in Rolling Shutter Cameras Ph.D. seminar talk – I Vijay Rengarajan EE11D035 Guides: Prof.A.N.Rajagopalan and Prof.R.Aravind September 23, 2016 Vijay Rengarajan A P, EE11D035, Image Processing and Computer Vision lab, Department of Electrical Engineering, IIT Madras

  2. Change Detection Find regions of changes between two images Aerial Imagery Aircraft hovering over an area to monitor changes Observed image compared with a reference image Scene is considered to be flat Main challenge: Camera motion

  3. Global Shutter Cameras All pixels expose at the same time Sensor plane t Exposure time, te 0 All pixels Exposure open Exposure close

  4. Rolling Shutter Cameras Exposure time of rows starts sequentially Sensor plane t Exposure time, te 0 Top row Bottom row Exposure open Total line delay, Td Exposure close

  5. Rolling Shutter Cameras Exposure during no camera motion Captured image Exposure during camera motion Captured image

  6. Rolling Shutter Cameras The type of distortion depends on the ratio of total line delay and exposure period Exposure 0 Top row Bottom row Camera motion Rolling shutter only (RS) Rolling shutter and motion blur (RSMB) Global shutter motion blur (GSMB)

  7. Spot the difference! Reference image Distorted image Registration of the two images Estimate camera motion for every row Detection of changes Ignore valid geometric and photometric changes Rolling shutter effect Change Motion blur

  8. Simple differencing will not work Reference image Distorted image Registration of the two images Estimate camera motion for every row Detection of changes Ignore valid geometric and photometric changes Reference image Rolling shutter effect Change Motion blur

  9. Simple differencing will not work Reference image Distorted image Registration of the two images Estimate camera motion for every row Detection of changes Ignore valid geometric and photometric changes Distorted image Rolling shutter effect Change Motion blur

  10. Simple differencing will not work Reference image Distorted image Registration of the two images Estimate camera motion for every row Detection of changes Ignore valid geometric and photometric changes Difference image Rolling shutter effect Change Motion blur

  11. Simple differencing will not work Reference image Distorted image Registration of the two images Estimate camera motion for every row Detection of changes Ignore valid geometric and photometric changes Detected changes Rolling shutter effect Change Motion blur

  12. Traditional works model global motion blur Blurred image Reference image Camera motion Exposure time O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, “Non-uniform deblurring for shaken images”, International Journal of Computer Vision, 2012.

  13. We introduce row-wise motion blur for rolling shutter Rolling shutter motion blur rows

  14. We introduce row-wise motion blur for rolling shutter Rolling shutter motion blur rows ith row exposure Discretised model Weighted sum of warped versions Pose space S

  15. We introduce row-wise motion blur for rolling shutter Rolling shutter motion blur rows ith row exposure Discretised model i+1 th row exposure Weighted sum of warped versions Pose space S

  16. We introduce row-wise motion blur for rolling shutter Rolling shutter motion blur rows Discretised model Weighted sum of warped versions Camera pose weight vector Matrix multiplication Distorted image row Rows of warped reference image How to model changes in addition to row-wise blur?

  17. Model change as an additive component Distorted row Blur-registered reference row Changes Camera pose weight vector Identity basis for changes Rows of warped reference image Distorted image row Change weight vector How to estimate and , given and ?

  18. Exploit sparsity of camera motion and changes 1 Data cost Photometric and geometric registration Priors Sparsity of camera motion Sparsity of changes Non-negativity of pose weights 1 2 2

  19. Results RSMB distorted image Registered image Detected changes Reference image

  20. Results Reference image RSMB image Registered image Detected changes Reference image

  21. Results Distorted image RSMB image Registered image Detected changes Reference image

  22. Results Registered image RSMB image Registered image Detected changes Reference image

  23. Results Detected changes RSMB image Registered image Detected changes Reference image

  24. Our algorithm estimates non-uniformly warped grid Reference image Estimated camera motion Registered image

  25. Results Registered image Reference image Distorted image Detected changes

  26. Results Registered image Reference image Distorted image Detected changes

  27. Comparison with sequential framework Global motion deblurring 2 RSMB image Deblurred RSMB image 1 Deblur and Register framework 1 Detected Changes Register rolling shutter effect 2 Reference image

  28. Comparison with sequential framework Global motion deblurring 2 RSMB image Deblurred RSMB image 1 Deblur and Register framework 1 Detected Changes Register rolling shutter effect 2 Reference image RS registered images Detected changes Liang et al. 2008 Our method Deblurred image Ringaby and Forssen 2012 Whyte et al. 2012 O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, “Non-uniform deblurring for shaken images”, International Journal of Computer Vision, 2012. C. Liang, L. Chang, and H. Chen, “Analysis and compensation of rolling shutter effect”, IEEE Transactions on Image Processing, 2008. E. Ringaby and P.E. Forssen, “Efficient video rectification and stabilisation for cell-phones”, International Journal of Computer Vision, 2012.

  29. Comparison with sequential framework 1 RSMB image 1 2 Register and Reblur framework RS-rectified image Register rolling shutter effect 2 Detected Changes Reference image Global motion blur registration

  30. Comparison with sequential framework 1 RSMB image 1 2 Register and Reblur framework RS-rectified image Register rolling shutter effect 2 Detected Changes Reference image Global motion blur registration Our method RS rectified image Reblurred image Detected changes Whyte et al. 2012 Grundmann et al.2012 M. Grundmann, V. Kwatra, D. Castro, and I. Essa, “Calibration-free rolling shutter removal”, International Conference on Computational Photography, 2012. O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, “Non-uniform deblurring for shaken images”, International Journal of Computer Vision, 2012.

  31. Comparison with RSMB deblurring framework Su and Heidrich (2015) deblur with the assumption of 2D parametric camera motion ignoring inplane rotations RSMB distorted image RSMB deblurred image Detected changes by Su and Heidrich Our method Reference image S. Su and W. Heidrich, “Rolling shutter motion deblurring”, IEEE Conference on Computer Vision and Pattern Recognition, 2015

  32. So far Model row-wise motion blur for rolling shutter cameras Model change as an additive component Jointly estimate camera motion and changes Only for flat 2D scenes!

  33. Aerial Imagery of 3D Scenes Imaging from drones -- prevalent Scene -- no longer flat Challenge: Motion registration at all depths

  34. Layered 3D Scene Model Rolling shutter motion blur model for 3D scene : Image from layer for camera pose : 2D Image from all layers for camera pose : For all camera poses :

  35. Motion of a 3D Scene Captured image at one camera pose

  36. Motion of a 3D Scene Images sensed during camera motion

  37. Motion of a 3D Scene Blurred image Different blur lengths at different layers

  38. Spot the difference! – 3D Scene Version Distorted image Reference image Registration of the two images Estimate camera motion for every row and at every depth layer Detection of changes Ignore valid geometric and photometric changes

  39. Change Detection for 3D Scenes 1 Background registration assuming flat scene The resultant changes include 3D objects and the actual changes Object filling Identify 3D object regions Layer registration Register the resultant changes at a scaled pose space of the background Regions that do not register at any relative depth correspond to the final change 2 3

  40. Change Detection for 3D Scenes 1 Background registration assuming flat scene The resultant changes include 3D objects and the actual changes Object filling Identify 3D object regions Layer registration Register the resultant changes at a scaled pose space of the background Regions that do not register at any relative depth correspond to the final change 2 3

  41. Change Detection for 3D Scenes 1 Background registration assuming flat scene The resultant changes include 3D objects and the actual changes Object filling Identify 3D object regions Layer registration Register the resultant changes at a scaled pose space of the background Regions that do not register at any relative depth correspond to the final change 2 3

  42. Change Detection for 3D Scenes 1 Background registration assuming flat scene The resultant changes include 3D objects and the actual changes Object filling Identify 3D object regions Layer registration Register the resultant changes at a scaled pose space of the background Regions that do not register at any relative depth correspond to the final change 2 3

  43. Scaled motion at different depth layers All depths see the same camera trajectory, but their motions on the image plane differ • Motion at image plane • For background • Translations and rotations • For layer ℓ • Translations multiplied by 1/dℓ • Rotations remain the same Camera dbg 1 dℓ<1 Relative depth of layer ℓ d

  44. Change Detection for 3D Scenes 1 Background registration assuming flat scene The resultant changes include 3D objects and the actual changes Object filling Identify 3D object regions Layer registration Register the resultant changes at a scaled motion of the background Regions that do not register at any relative depth correspond to the final change 2 3

  45. Change Detection for 3D Scenes - Example Clean image

  46. Change Detection for 3D Scenes - Example Ground-truth depth map The lighter the gray, the closer to the camera

  47. Change Detection for 3D Scenes - Example RSMB distorted image

  48. Change Detection for 3D Scenes - Example Reference image

  49. Change Detection for 3D Scenes - Example RSMB distorted image

  50. Change Detection for 3D Scenes - Example 1 Background registered image

More Related