Digital Imaging and Remote Sensing Laboratory

1. Automatic Tie-Point and Wire-frame Generation From Oblique Aerial Imagery Seth Weith-Glushko Advisor: Carl Salvaggio Research Proposal November 7, 2003 Digital Imaging and Remote Sensing Laboratory

2. Table of Contents • The Problem • Specific Aims • Proposed Solution • A description of the individual transforms and algorithms used to make up the proposed algorithms • Methods • Timetable • Budget • References Digital Imaging and Remote Sensing Laboratory

3. The Problem • Using photogrammetric techniques, tie matching algorithms and bundle adjustment algorithms, it is possible to create a 3D model using a series of images of an object • These images must be around an axis (for objects) or ortho-rectified (for landscapes) for the algorithms to work • We want to be able to use oblique imagery for input into the algorithms • By using oblique imagery, more pixels are available to describe the features of a 3D object than would be in an ortho-rectified image Digital Imaging and Remote Sensing Laboratory

4. Specific Aims • Current algorithms only work on ortho-graphic (for tie-points) and axial imagery (for wire-frame generation) • The goal is to draw from both photogrammetry and computer graphics to develop two algorithms which can work in unison • A tie-point algorithm that works on oblique imagery • A bundle adjustment algorithm that can use oblique imagery • In the future, this algorithm would become part of a suite of algorithms that could generate an accurate 3D model of scene independent of the type of imagery used Digital Imaging and Remote Sensing Laboratory

5. Proposed Solution • The algorithm below relies heavily on the use of INS (Inertial Navigational System) data • The input would be a series of oblique images made around a common area • The output would be a matrix of matching points and a file using a common 3D format Input: A series of oblique images around a common area Ortho-rectification Transform Image Processing Converts an oblique image into an ortho-rectified image • Histogram processing Digital Imaging and Remote Sensing Laboratory

6. Proposed Solution Definition of Points Point matching algorithm Geometric Transform Use the Laplacian of Gaussian (LoG) filter to find points of high frequency (i.e. edges) Use point distance comparison, point scale comparison and point angle comparison to find matching points Using matched points, generate a geometric transformation and use registration as indicator of “goodness” of points Digital Imaging and Remote Sensing Laboratory

7. Proposed Solution Inverse Geometric Transform Inverse Orthorectification Transform Bundle Adjustment Algorithm Perform an inverse geometric transformation using previous transformation matrix Convert the orthorectified image back into its original oblique form Using pairs of matched points, define points in 3D space Output: Matrix containing matched pairs between images Digital Imaging and Remote Sensing Laboratory

8. Proposed Solution Output: 3D wire-frame mesh of a scene Interface with 3D System Using software libraries and defined 3D points, generate an output file Digital Imaging and Remote Sensing Laboratory

9. Ortho-rectification Transform • The ortho-rectification transform converts an oblique image into an ortho-rectified (flat) image by means of a linear equation • There are four unknowns. We can use the fiducial points of an image as the four points we need to solve for the constants Digital Imaging and Remote Sensing Laboratory

10. Image Processing • Image processing needs to be performed because images with dissimilar digital count affect the point generation operator • Histogram matching is performed to minimize this dissimilarity Images courtesy C. Salvaggio Digital Imaging and Remote Sensing Laboratory

11. Definition of Points • To define points the Laplacian of Gaussian operator is used • Walli found that if there is an edge in an image, a thresholded filtered image would show a point at that edge Images courtesy K. Walli Digital Imaging and Remote Sensing Laboratory

12. 25% Point Matching Algorithms • To match defined points, three algorithms are primarily used: pixel distance match, scale match, angle match • Another matching criteria is LoG maxima similarity Digital Imaging and Remote Sensing Laboratory

13. Pixel Distance Match • This algorithm works by comparing distances between pixels in a matrix 3 2 1 2 3 4 3 1 2 3 4 2 1 1 1 2 3 4 1 2 3 4 # Same Elements: 2 3 2 Distance= Digital Imaging and Remote Sensing Laboratory

14. Scale Match • This algorithm works by comparing ratios of distances between pixels in a matrix 3 1 2 3 4 2 1 2 3 4 3 2 1 1 0 1 2 3 4 5 6 7 0 1 2 3 4 Distance= Compare Ratios= Ratios of distances from like points is equal! Digital Imaging and Remote Sensing Laboratory

15. 3 2 Vertice 1 Angle Match • This algorithm works by comparing the angle formed by the triangle of 3 pixels in a matrix 3 2 1 2 3 4 3 1 2 3 4 2 1 1 1 2 3 4 1 2 3 4 Point Set 1 3 2 Point Set 2 Vertice 1 Digital Imaging and Remote Sensing Laboratory

16. Geometric Transformation • Using the matched pixels, solutions to the affine polynomial problem are found • Using the affine polynomial, one ortho-rectified image is geometrically transformed so that it can register with another ortho-rectified image. • Quality metrics are performed to determine whether the registration is good. Hence, the matched points are good. Digital Imaging and Remote Sensing Laboratory

17. Inverse Transformations • The matched points are put through an inverse geometric transformation and an inverse ortho-rectification transform to return the points to their original oblique pixel form • A matrix of matched points is output Digital Imaging and Remote Sensing Laboratory

18. Bundle Adjustment Algorithm • Bundle adjustment algorithms allow the mapping of 2D points into 3D space using more than two images around a common point • The algorithm estimates the underlying plane geometry of a scene Images courtesy M. Pollefeys Digital Imaging and Remote Sensing Laboratory

19. 3D Library Interfacing • Using these 3D points generated from the bundle adjustment algorithm, a triangle mesh is created which forms the structure of the wire-frame scene • Also, a texture map is generated from bundle adjustment. This map is overlaid on the mesh • The full model is saved to a generic 3D format Digital Imaging and Remote Sensing Laboratory

20. Methods • Using a programming environment, engineering code will be developed to determine the feasibility of this algorithm • If it is feasible, quality metrics will be applied to determine effectiveness • Visual analysis, absolute mean variance, and deviation from a polynomial model (RMSDE) can be used to check tie-point generation • Visual analysis and post-photogrammetric analysis can be used to check wire-frame generation Digital Imaging and Remote Sensing Laboratory

21. Timetable • September 1, 2003 – November 15, 2003 • Search for previous research, background knowledge • November 15, 2003 – April 1, 2004 • Development of algorithm and engineering code • April 1, 2004 – May 15, 2004 • Complete paper, poster and presentation Digital Imaging and Remote Sensing Laboratory

22. Budget • No money will be required for this project as the investigator has all of the resources he currently requires • 2 credits will be required for both Winter and Spring Quarters • Due to the nature of the contract, most of the work performed will be done on a pay basis • Flexibility in the experimenter’s schedule was required Digital Imaging and Remote Sensing Laboratory

23. References • Honkavaara, Eija and Anton Hogholen. “Automatic Tie Point Extraction in Aerial Triangulation.” International Society for Photogrammetry and Remote Sensing, 16th Congress, Vienna, July 1996. 337-342. • Moffitt, Francis H. and Edward M. Mikhail. Photogrammetry. 3rd Ed. New York: Harper and Row, 1980. • Pollefeys, M. “3D Geometry from Images.” 15 Oct. 2003. <http://www.esat.kuleuven.ac.be/~pollefey/tutorial/tutorialECCV.html> • Walli, Karl C. “Multisensor Image Registration Utilizing the LOG Filter and FWT.” Diss. Rochester Institute of Technology, 2003. • Wolf, Paul R. Elements of Photogrammetry. 2nd Ed. New York: McGraw-Hill, 1983. Digital Imaging and Remote Sensing Laboratory

24. Questions Are there any? Digital Imaging and Remote Sensing Laboratory