1 / 25

Generating Seamless Stereo Mosaics from Aerial Video

Generating Seamless Stereo Mosaics from Aerial Video. Zhigang Zhu Allen R. Hanson, Harpal S. Bassali Howard Schultz, Edward M. Riseman Computer Vision Lab Computer Science Department University of Massachusetts at Amherst zhu@cs.umass.edu http://www.cs.umass.edu/~zhu. Introduction.

kay-robinson
Download Presentation

Generating Seamless Stereo Mosaics from Aerial Video

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Generating Seamless Stereo Mosaics from Aerial Video Zhigang Zhu Allen R. Hanson, Harpal S. Bassali Howard Schultz, Edward M. Riseman Computer Vision Lab Computer Science Department University of Massachusetts at Amherst zhu@cs.umass.edu http://www.cs.umass.edu/~zhu

  2. Introduction • Objectives • Develop methods to automatically generate geo-referenced stereo mosaics from video sequences • Definitions • Free Mosaic: composite of video sequence by registering overlapping frames • subject to drift relative to the terrain • constrains: pure rotation or planar scenes • Geo-Mosaic: use 3D instrumentation to constrain mosaic to world coordinates • Stereo Mosaics : a pair of mosaics from a single camera • seamless under motion parallax • preserve 3D information • can be viewed in 3D directly

  3. Important Issues in 3D Video Mosaicing • Representation • compact representation for a large-scale 3D scene • orthogonal (DEM), perspective (mosaic), parallel-perspective • Computation • how expensive are the computations of the algorithms? • Goal: affordable, efficient and robust mosaicing • Accuracy • accurate for 3D viewing and 3D reconstruction ?

  4. Sensor Image Plane “Right” Mosaic “Left” Mosaic Geometry, Representation and Properties • Sensor motion is pure translation

  5. Re-Organizing the images….

  6. GPS Height H from Laser Profiler baseline displacement P(X,Y,Z) disparity Fixed ! Two views from different perspective stereo Recovering Depth from Mosaics • parallel-perspective stereo mosaics • Depth accuracy independent of depth

  7. a b c Stereo mosaics of Amazon rain forest • 166-frame telephoto video sequence -> 7056*944 mosaics

  8. Stereo viewing • Red: Right view; Blue/Green: Left view

  9. Computation: how expensive in in real world application? • Pro-processing • Arbitrary motion other than a 1D translation / 3D translation • Camera orientation estimation and image rectification • Mosaicing • Image sequence is not dense enough for seamless mosaics • how to generate parallel-perspective projection ? • 3D recovery • how expensive is the match in stereo mosaics? • Baseline and epipolar geometry of stereo mosaics • Post-processing • parallel-perspective 3D mosaics to an orthogonal DEM

  10. 3D path of the camera: 3D rotation + 3D translation Dominant motion direction Original image frames Path of the camera: 3D translation Dominant motion direction Y Rectified image frames Step 1. Pro-ProcessingMotion estimation & rectification • Camera pose estimation using navigation instrumentation and bundle adjustment • only sparse tie points widely distributed in the two images are needed • Image rectification • transformation on two narrow slices in each frame

  11. Rear slit Front slit : Both slits are sub-images of m columns (m>=1) Perspective image Rays of left view Rays of right view Multi-perspective mosaics (1) Right view mosaic Left view mosaic …… …… …… …… View interpolation Ray interpolation Rays of left view Rays of right view Parallel-perspective mosaics (2) Right view mosaic Left view mosaic Step 2. 3D Mosaicingseamless mosaicing with motion parallax In a multi-perspective projection mosaic, each sub-image is full perspective, but sub-images from different frames will have different viewpoints. This may cause seams in the mosaic due to motion parallax. • Geometric Seams - • Clearly Visible to Human Eyes, especially along depth boundaries • Introduce Error in Height Estimation

  12. (Tx+Sx, Ty+Sy) (Txi,Tyi) (Tx, Ty) 2nd fixed line (x2, y2) (x1, y1) 1st fixed line y0= dy/2 IP of 2nd fixed line (xi, yi) IP of 1st fixed line IP of interpolated fixed line (X,Y,Z) PRISM parallel ray interpolation for stereo mosaicing Interpolated view: Mosaic coordinates: - Take a slice of certain width from each frame - Perform local registration between the overlapping slices - Generate parallel interpolated views between two known views - Re-project the point back to the mosaic

  13. 2D mosaic from sparse image sequence ( 40-pixel interframe motion) 2D mosaic from dense image sequence ( 4-pixel interframe motion) 3D mosaic from sparse image sequence ( 40-pixel interframe motion) Comparison : 2D mosaic & 3D mosaic

  14. xl txl(yl) left mosaic (xl,yl) yl dy/2 xr txr(yr) (xl,yl) right mosaic yr (xr,yr) txr(yr)- txl(yl) dy/2 Step 3. 3D reconstructionepipolar geometry of stereo mosaics • Epipolar curve • 1D search • Near horizontal line • Coverged pair • small search region • epipolar curve

  15. Epipolar Geometry in Real Stereo mosaics “Left” Mosaic

  16. Epipolar Geometry in Real Stereo mosaics “Right” Mosaic

  17. Epipolar Geometry in Real Stereo mosaics Depth Map: Brighter is higher elevation

  18. Step 4. Post-Processing from 3D mosaic to DEM • Just a transformation !! • (X,Y,Z) world coordinates • (xl,yl) left mosaic coordinates • (xr,yr) right mosaic coordinates • H : a reference height (depth) • F: focal length of the camera • dy: distance between left and right slits • by: adaptive baseline • Dy = yr - yl

  19. 3D Rendering Result

  20. Motion Refinement for Geo-Mosaic- when geo-data is not accurate Ground Truth Geo Meaursement - absolute error EKF Esimation • Extended Kalman Filter (EKF) approach Flying path Unconstrained -Image Match -Accumulating error Time Update “Predict” from image registration Measurement Update “correct” by geo-data Motion parameter A: Warp matrix t : translation

  21. Geo-mosaic Free Mosaic Geo-Reference Mosaicsframe-by frame mosaicing

  22. Free mosaic Frame-by-frame geo-mosaic curve diamond Geo-mosaic from free mosaic straight square Geo-Reference Mosaicsglobal warping from free mosaic

  23. Methods Summarydistribute the computations in four steps • Pro-processing • Motion estimation: sparse tie points distirbuted in entire frames • Rectification and Mosaicing (PRISM) • Process two narrow slices • 3D recovery (Terrest) • stereo match only in two mosaics • Post-processing • just a coordinate transformation

  24. Accuracy of 3D from Stereo Mosaics • Adaptive baselines and fixed disparity - uniform depth resolution • Ray interpolation between two successive views is similar to image rectification • 3D recovery accuracy is comparable to that of a perspective stereo with an optimal baseline

  25. Next Steps • Camera calibration and bundle adjustments • Geometric and Photometric Seamlessness • Using Structure Information and video –photo matching • Error analysis in geo-referenced mosaic and 3D reconstruction using ground truth

More Related