Ray Divergence-Based Bundle Adjustment Conditioning for Multi-View Stereo

1. Ray Divergence-Based Bundle Adjustment Conditioning for Multi-View Stereo Mauricio Hess-Flores1, Daniel Knoblauch2, Mark A. Duchaineau3, Kenneth I. Joy4, FalkoKuester5 1,4Institute for Data Analysis and Visualization, University of California - Davis, Davis, CA, USA 2,5University of California - San Diego, San Diego, USA 3Lawrence Livermore National Laboratory, Livermore, CA, USA 1mhessf@ucdavis.edu, 2dknoblau@ucsd.edu, 3duchaine@llnl.gov, 4joy@cs.ucdavis.edu, 5fkuester@ucsd.edu Abstract- An algorithm that shows how ray divergence in multi-view stereo scene reconstruction can be used towards improving bundle adjustment weighting and conditioning is presented. Starting with a set of feature tracks, ray divergence when attempting to compute scene structure for each track is first obtained. Assuming accurate feature matching, ray divergence reveals mainly camera parameter estimation inaccuracies. Due to its smooth variation across neighboring feature tracks, from its histogram a set of weights can be computed that can be used in bundle adjustment to improve its convergence properties. It is proven that this novel weighting scheme results in lower reprojection errors and faster processing times than others such as image feature covariances, making it very suitable in general for applications involving multi-view pose and structure estimation. Introduction Algorithm Results (continued) Ray divergence computation • Our weighting scheme provides the best combination of final reprojectionerror, processing time and computational complexity. • Beder’salgorithm (UWBA and UIBA) involves matrix inversion and SVD, while the feature covariance method CBArequires multiple exponential evaluations for each covariance matrix. • Multi-view scene reconstruction is currently a prominent research area, with many applications. • Despiteaccurate feature matching, reconstruction accuracy relies on accurate camera calibration. • In the absence of ground-truth, algorithms usually resort to bundle adjustment to reduce reprojection error, which is very expensive and needs a good starting point. • Main contribution: show how simple reconstruction ray divergenceis an inexpensive yet powerful tool that can aid in bundle adjustment convergence. • Ray divergence:shortest distance between rays from each respective camera center Ciand through each pixel position of a given feature match. • For SIFT feature matches between two images, eliminate epipolar geometry outliers with RANSAC. • Compute ray divergence per inlier feature match. • ‘Good’ baselines are necessary, such as those obtained through frame decimation. Ray divergence histogram Ray divergences vary smoothly – their histogram can be modeled as a Gaussian probability density function. Input images Ray divergence ‘d’ per feature match • IterationsI, final total reprojection error R (pixels) and processing time t(seconds) in (I,R,t) format for the four different weighting schemes: Ray divergence histogram (x1,y1) (x2,y2) Weighted bundle adjustment (x2,y2) • Bundle adjustment (a) minimizes reprojection error by solving the normal equations (b)at each iteration. • Gaussian pdf values for each feature match are input to weighted bundle adjustment (c)as 2x2 covariance matrices into the diagonal matrix ∑x. • High divergences correspond to lower pdf values and are ‘penalized’ as lower weights, thus improving bundle adjustment convergence. (x1,y1) Features Features Ki = intrinsics Ri = absolute rotation ti = parametric distance Ray directions: Closest-distance points on each ray: Assuming accurate feature matching, ray divergence varies smoothly, and is a function of all camera intrinsic, extrinsic and radial distortion inaccuracies. (a) • Reconstructions with our method: original point clouds (left) and adding Furukawa’s PMVS (right): (b) (c) Stockton dataset Dense, smooth ray divergence surfaces Multi-view extension Ray divergence correlates well with Beder et al.’s confidence measure for scene structure1, but is cheaper to compute and includes radial distortion. In sequential multi-view reconstruction, for each feature in a new image we assign the Gaussian-based weight corresponding to the ray divergence statistics for the feature matches to the prior image. Medusa dataset Results Megascene1 dataset Processing times and reprojection errors obtained for weighted bundle adjustment using ray divergences (RDBA) was compared versus other weighting schemes: image feature covariances (CBA) and reconstruction confidence1with (UWBA) and without (UIBA) including image feature covariances. Ray divergences – Palmdale image pair. Reconstruction confidence measure1– Palmdale image pair. Encodes errors at all stages. Palmdaledataset Lighter colors = higher values 1Beder, C. and Steffen, R. Determining an initial image pair for fixing the scale of a 3d reconstruction from an image sequence. In: DAGM Symposium 2006. Berlin, Germany 2006, S. 657-666. LLNL-POST-510151 This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.