1 / 74

New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation

New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation. Jin Zhou June 16 th , 2009 Dissertation Defense. Outline. Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration

kareem
Download Presentation

New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation Jin Zhou June 16th, 2009 Dissertation Defense

  2. Outline • Introduction • Rectification based on Virtual Sequential Rotation • Image Rectification for Stereoscopic Visualization • Camera Calibration • Stereoscopic View Synthesis from Monocular Endoscopic Sequences • Rapid Cones and Cylinders Modeling from Single Images • Robot Vision • Conclusions and Future Work

  3. The Geometry of 3D to 2D Images are 2D projections of the 3D world

  4. 3D Vision – The Problem How do we extract 3D information from 2D images? ? 3D of the objects ? ? ? ? 3D of the cameras

  5. 3D Vision – Applications • Augmented Reality • Iron Man, Lord of Rings, Kingkong • Scene Modeling • Google SketchUp, Autodesk ImageModeler • Virtual Touring • Microsoft PhotoSynth • 3D Visualization • 3D Gaming, 3D Movies, 3D Medical Images • Robots

  6. 3D Vision – A Human Perspective • Size • Linear Perspective • Object Connections • Stereo • Motion • Occlusion • Shading

  7. 3D Vision – Computational Approaches • Different approaches use different cues • Different problems require different approaches. • Structure from Motion (SfM) • Rely on point correspondences • Single View Based Modeling (SVBM) • Rely on knowledge of the scene • Camera calibration • All approaches require the images are calibarated first (either manual or automatic)

  8. 3D Vision – Practical Challenges • Camera information is often unavailable • Point correspondences are not reliable and computationally costly to compute • Image resolution is limited in many applications • Degeneracy

  9. 3D Vision – Limitations of Current Approaches • Distortion • Degeneracy • Due to high degree of freedom of geometric models • Lack of geometric meaning • Most approaches are purely based on algebraic derivations or imaginary objects. • Not accurate or not convenient (SVBM).

  10. 3D Vision – Our Contributions • Novel image rectification schemes are proposed based on sequential virtual rotation • Novel approaches are proposed for the following problems • Image rectification for stereoscopic visualization • Camera calibration • Stereoscopic view synthesis from monocular endoscopic sequences • Rapid cones and cylinders modeling • Monocular-vision-guided mobile robot navigation

  11. 3D Vision – Results of Our Approaches • No affine/projective distortion • Can handle degeneracy • Intuitive geometric meanings • Lead to useful insights for some practical problems • Accurate and fast

  12. Publications & Manuscripts • Journals • Jin Zhou and Baoxin Li, "Image Rectification for Stereoscopic Visualization", Journal of the Optical Society of America A (JOSA), Vol. 25, No. 11, pp. 2721-2733, 2008. • Wenfeng Li, Jin Zhou, Baoxin Li, and M. Ibrahim Sezan, " Virtual View Specification and Synthesis for Free Viewpoint Television", IEEE Transactions on Circuit and Systems for Video Technologies , Vol, 19, No. 4, pp.533-546, 2009. • Jin Zhou and Baoxin Li, "Rapid Cones and Cylinders Modeling from a Single Calibrated Image Using Minimal 2D Control Points", submitted to Machine Vision and Applications (under revision) • Jin Zhou and Baoxin Li, “Stereoscopic View Synthesis from Monocular Endoscopic Image Sequences“,(manuscript under internal revision; to be submitted to IEEE Transactions on Medical Imaging by the end of June 2009) • Jin Zhou and Baoxin Li, “Image Rectification Based on Infinity Homography”, (manuscript under internal revision; to be submitted to IEEE Transactions on Image Processing by the end of June 2009) • Refereed Conference papers • Jin Zhou, Ananya Das, Feng Li, Baoxin Li, "Circular Generalized Cylinder Fitting for 3D Reconstruction in Endoscopic Imaging Based on MRF", In 9th IEEE Computer Society Workshop on Mathematical Methods in Biomedical Image Analysis (Joint with CVPR 2008). • Jin Zhou and Baoxin Li, "A Four Point Algorithm for Fast Metric Cone Reconstruction from a Calibrated Image", In 4th International Symposium on Visual Computing (ISVC), 2008. • Xiaolong Zhang, Jin Zhou and Baoxin Li, "Robust Two-view External Calibration by Combining Lines and Scale Invariant Point features", In 4th International Symposium on Visual Computing (ISVC) 2008.

  13. Publications (Cont’d) • Jin Zhou and Baoxin Li, “Exploiting Vertical Lines in Vision-Based Navigation for Mobile Robot Platforms”, International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2007. • Xiaokun Li, Roger Xu, Jin Zhou, Baoxin Li, "Creating Stereoscopic (3D) Video from a 2D Monocular Video Stream", In 3rd International Symposium on Visual Computing (ISVC), 2007. • Wenfeng Li, Jin Zhou, Baoxin Li, M. Ibrahim Sezan, "Virtual View Specification and Synthesis in Free Viewpoint Television Application", 3D Data Processing, Visualization and Transmission (3DPVT), 2006. • Jin Zhou and Baoxin Li, "Image Rectification for Stereoscopic Visualization without 3D Glasses", ACM International Conference on Image and Video Retrieval (CIVR), 2006. • Jin Zhou and Baoxin Li, “Homography-based Ground Detection for a Mobile Robot Platform using a Single Camera”, International Conference on Robotics and Automation (ICRA), 2006. • Jin Zhou and Baoxin Li, “Robust Ground Plane Detection with Normalized Homography in Monocular Sequences from a Robot Platform”, International Conference on Image Processing (ICIP), 2006. • Jin Zhou and Baoxin Li, “Rectification with Intersecting Optical Axes for Stereoscopic Visualization”, International Conference on Pattern Recognition (ICPR), 2006.

  14. Outline Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single Images Robot Vision Conclusions and Future Work

  15. What is Image Rectification? Image rectification is a process to transform the original images to new images which have desired properties.

  16. General Image Transformations H? Image transformation can be defined by a 3x3 matrix H, which is called Homography.

  17. Image Rectification based on Virtual Rotation • Homography of camera rotation/zooming • If we normalize the coordinates • Assume R = I Camera orientation is determined at the same time

  18. Advantages of the New Rectification Schemes • Intuitive geometric meaning • Robust • Rotation parameters can be computed by various basic image features, such as points, lines and circles. • Can be used for camera calibration. • Can be used for 3D information extraction. • Lead to non-distorted results • Reason: Rotation do not introduce affine/projective distortion

  19. Rotations are Decomposed as Euler Angles pitch yaw roll

  20. Rotation Parameters Can Be Estimated by Basic Image Features • Each rotation has only one degree of freedom and thus only needs one constraint. • Example: transforming a point on to y axis Normalize Ambiguity!

  21. Outline Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single Images Robot Vision Conclusions and Future Work

  22. Image Rectification for Stereoscopic (3D) Visualization Nielsen Consumer Insight2009 Industry Outlook: Cinema admissions will increase fueled by growing availability of 3D movies and theatres. Deloitte: 3D becomes an obligation, not an option, at the movies The Principle of Stereoscopic (3D) Visualization

  23. The Problem general camera setup stereo camera setup Given two arbitrary images, rectify them so that the results look like a stereo pair.

  24. An Example

  25. Motivation • Stereo content is scarce • Stereo cameras/camcorders are expensive • Common users seldom use stereo cameras/camcorders • Price of a current 3D Display is affordable • We want to generate stereo content from images/videos taken by common cameras

  26. Our Approach – Rectification based on Virtual Rotation We can “rotate” cameras to standard stereo setup.

  27. Calibrated Case • Constraints of the stereo camera pair: • The two cameras have the same intrinsic parameters (K) and orientation (R) • The camera’s optical axis is perpendicular to the baseline (C1 – C2) • i.e. the camera’s x axis has the same direction with the baseline UnKnown K1 Known Any vector

  28. Uncalibrated Case • For the uncalibrated case, all K, R and C are unknown. We can only start from the fundamental matrix and point correspondences. • Estimate H2 (homography for the second image) ?

  29. Determine R based on Sequential Virtual Rotation Constraints: In stereo pair, the epipole is at horizontal infinity First rotate around z axis so that the point is transformed to x axis (i.e. y = 0) Rotate around y axis so that the point is transformed to infinity.

  30. Estimate H1

  31. Estimate H1 • Determine a, b and c • Property of standard stereo setup: • For two points with the same depth, their projection on different images should have the same distance (Points with the same depth should have the same disparity). • Approach • Group points by similar disparities • Then compute a, b by minimizing

  32. Results Shear distortion Hartley’s Method Our Method Original Pair

  33. Results Shrink horizontally Hartley’s Method Our Method Original Pair

  34. Outline Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single Images Robot Vision Conclusions and Future Work

  35. Camera Auto-Calibration from the Fundamental Matrix Traditional Approaches: Kruppa Equations dual image of the absolute conic Huang-Faugeras constraints Cons: Complex and hard to understand! Derivation for degenerate cases are purely algebraic.

  36. Our Approach We transform the original pair to a standard stereo pair through sequential virtual rotation and zooming 7 DOFs 7 Parameters

  37. Decomposition Illustration

  38. Solving The Equation

  39. Degeneracy Analysis

  40. Degeneracy Analysis

  41. Results of Monte Carlo Simulation

  42. Outline Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single Images Robot Vision Conclusions and Future Work

  43. Stereoscopic View Synthesis From Monocular Endoscopic Videos 3D imaging helps to enable faster and safer surgical operations Two-view image rectification cannot be applied to the new problem

  44. The Framework • We proved: • Affine 3D reconstruction is sufficient. • Linear interpolation in normalized disparity field is equal to linear interpolation in 3D space. • Challenges: • Image quality is poor • Degeneracy

  45. Strategy for Solving Degeneracy We assume the initial two frames have same orientation (i.e. they are rectified) The assumption makes the DOF of the fundamental matrix from 8 to 2! No Assumption Degeneracy! Assume the two frame are rectified

  46. Interpolation a) Shows the disparities based on the SfM results b) We do Delaunay triangulation and interpolate each triangle c) We pick a set of grid points from b) and do bilinear interpolation d) We fill holes using Laplacian interpolation and do smoothing.

  47. Results of Synthetic Data Disparity image after triangulation Final disparity image Ground truth Stereo images

  48. Results of Real Data fryover creel gray

  49. Outline Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single Images Robot Vision Conclusions and Future Work

  50. Rapid Cylinders and Cones Modeling from A Single Image

More Related