Distortion Correction ECE 6276 Project Review

1. Distortion CorrectionECE 6276Project Review Team 5: Basit Memon Foti Kacani Jason Haedt Jin Joo Lee Peter Karasev

2. Outline • Motivation • Components • Component Optimization • Current Results • Plans for Catapult C • Schedule

3. Objective Given a distorted image with known size and known lens distortion parameter, generate an undistorted image.

4. Motivation – Why? • The formation of undistorted images can be described by a series of matrix multiplications • Distortion makes it very difficult to calibrate a camera to measure geometry (depth, size, orientation, etc) • Many applications in image processing and computer vision like structure estimation, image mosaicing, and ultimately vision-based control.

5. Motivation contd.. Application: Measure motion and geometry Problem: Known geometry in the scene is warped, relationship between 3D and 2D points is nonlinear. Solution: Undo the distortion, so x2D = A * X3D

6. Literature Review (I) K.T. Gribbon, C.T. Johnston, and D.G. Bailey, “A Real-time FPGA Implementation of a Barrel Distortion Correction Algorithm with Bilinear Interpolation” • Focus on reducing hardware complexity. • Uses LUTs to store mapping data. • No quantitative results provided. • Logic resource utilization on RC-100 is 51%

7. Literature Review (II) • Qiang, L.; Allinson, N.M.;” FPGA Implementation of Pipelined Architecture for Optical Imaging Distortion Correction”, Signal Processing Systems Design and Implementation, 2006. SIPS '06. • Same algorithm as previous one • Implementation on a Xilinx FPGA XCS3 1000-4 uses 75% of the hardware multipliers. • Residual error of the undistorted image was 1.5% of the distorted image.

8. Literature Review (III) • Hany Farid & Alin C. Popescu, “Blind Removal of Lens Distortion”, Journal of the Optical Society of America 2001 • For removal of distortion in absence of any calibration data. • Uses polyspectral analysis to detect higher order correlations in frequency domain which are proportional to the distortion. • Computationally intensive. • No quantitative results. • Accuracy is not comparable to those based on known distortion parameters.

9. Components to Achieve Objective Component 1: Matlab forward distortion function • Verifies correctness of undistortion algorithm Component 2: Data ordering test bench • Order the C++ input stream from MATLAB generated data Component 3: Undistortion lookup table or lookup function • Initial prototype in MATLAB Component 4: Least squares interpolation lookup function • Initial prototype in MATLAB • Compare different techniques such as LUT vs NEAREST NEIGHBOR Component 5: Verification structure • Compare original C++ result to undisorted image in MATLAB MATLAB C++  Catapult C  MATLAB

10. Component Optimization Questions • What size buffers do we need to compare against previous frames? •  For undistortion function can we compute them dynamically or do we need a pre-defined LUT? •  For finding fast least squares / linear system solver, compare speed cost vs. nearest neighbor and effect on output error

11. Matlab Demo of Algorithm - Original

12. Matlab Demo of Algorithm - Distorted

13. Matlab Demo of Algorithm - Recovered

14. Design Goals • Limited Buffers • 8 bits per sub-pixel (24 bits total) • Resolution (up to 640x480) • Concerned with geometry • Area, Throughput, Latency • Target a low-cost implementation that handles consumer video application pixel clocks of 165 MHz (Apprx 6 ns cycle time).

15. Test Vectors The Line Test Original Image Distorted Image Recovered Image

16. Plans for Catapult C Code • Use C/C++ and Algorithmic C data types to describe synthesizable hardware • Architectures (type of hardware interface (streaming buffers) ) • Constraints (Throughput, area, latency) • RTL generation and verification • Optimizations -Pipelining -Parallelism -Loop Unrolling -Scheduling -Streaming buffers/ Read & Write

17. Project Timeline

18. Project Risks Risk 1: Slowness of interpolation plan: the key goal of undistortion is “geometrical accuracy” which hold even if noise is injected. To mitigate this risk we have a forward mapping algorithm that is fast compared to any interpolation method but at the cost of missing pixels near the edges. Risk 2: Not enough storage space to keep a lookup table of coordinates. To mitigate this risk we could compute coordinates on the fly at the cost of math operations. Risk 3: Cannot find least squares method for FPGA. To mitigate this risk we could do nearest neighbor interpolation.

19. Current Status • Straight lines are recovered with minimal injection of missing points or noise through our algorithm. • Add more quantitative results on the geometrical accuracy of recovered images than presented in previous results. • Generalize the coordinate mapping to make the implementation more robust to varying  distortion models and hence supporting more cameras “on the fly.”

20. References • Yi Ma, Stefano Soatto, et al., “An Invitation to 3-D Vision” • Richard Hartley, Andrew Zisserman, “Multiple View Geometry in Computer Vision” • Edward M. Mikhail, James S. Bethel, J. Chris McGlone, “Introduction to Modern Photogrammetry”

21. Questions? ?