Distortion Correction ECE 6276 Project Review

1. Distortion CorrectionECE 6276Project Review Team 5: Basit Memon Foti Kacani Jason Haedt Jin Joo Lee Peter Karasev

2. Outline • Objective/Motivation • Movement of Data • C code • Conversion to Catapult C • Initial Results • Current Issues • Plans for Optimization • Schedule • Meeting with Mike Bradley

3. Objective Given a distorted image with known size and known lens distortion parameter, generate an undistorted image.

4. Motivation – Why? • The formation of undistorted images can be described by a series of matrix multiplications • Distortion makes it very difficult to calibrate a camera to measure geometry (depth, size, orientation, etc) • Many applications in image processing and computer vision like structure estimation, image mosaicing, and ultimately vision-based control.

5. Motivation contd.. Application: Measure motion and geometry Problem: Known geometry in the scene is warped, relationship between 3D and 2D points is nonlinear. Solution: Undo the distortion, so x2D = A * X3D

6. Movement of Data Input Output Algorithm Input Buffer RAM on Chip 1 Pixel/Line Random Read LUT

7. C Code Reference Design I/O scheme: Write to output in raster order, read input randomly (From buffer). New input put into buffer in raster order. Coordinate in buffer does not have closed form expression- approximate with power series created in matlab, store as lookup table. const double powercoeffs[256][5] = { 0.0001676486936455572,0.9984675943966409,0.004271662030922045,-0.004648466894072161,0.001731029514912795, 0.0001678607731736833,1.004564329741083,0.004318757090371574,-0.008746108062078511,0.001856610695386462,… Top-level Entry point- block of pixels in and out, reset, index of distortion level void undistort_ref( const unsigned char pixels_in[PIXELS], unsigned char pixels_out[PIXELS], bool reset, unsigned short kappa_idx ); if( (iid >= 0) && (jjd >= 0) && (iid < HEIGHT) && (jjd < WIDTH) ) pixels_out[ i_*WIDTH + j_ ] = pixels_in[ (iid*WIDTH) + jjd ]; else pixels_out[ i_*WIDTH + j_ ] = 0; Conditional Write operation (ensure in-bounds) // read next frame n = fread(pix_in, 1, PIXELS, fin); // do the work for this block undistort_ref( pix_in, pix_out, 0, kappa_idx); // write what we undistorted n = fwrite( pix_out, 1, PIXELS, fout ); Code verification- load binary input vectors, run, save, compare to matlab generated binary files

8. Conversion to Catapult C Catapult C does not support dynamic memory allocation - use static arrays for memory - simplify C code structure to avoid pointers Any value that needs to be stored must be declared as static or could be synthesized out Top-level default instantiation assumes that input and output ports are wires, must explicitly declare memories Casting for printing is not automatically supported, must use .to_int() and .to_double() functions Had to simplify C code structure to not use as many pointers as concept is not as well defined in Catapult C Verify correctness in Visual Studio and Catapult environment

9. Test Vector – Original Image

10. Test Vector – Distorted Image

11. Test Vectors – Recovered Image

12. Initial Catapult C Synthesis (1/3)

13. Initial Catapult C Synthesis (2/3)

14. Initial Catapult C Synthesis (3/3)

15. Current Problems • Top issue is Area – we can’t support large image sizes based on current algorithm implementation. • How do we achieve the ability to stream images in and then pass through our algorithm? Need to segment our algorithm and stream in portions of a larger image. • Need to allocate large arrays to RAMs, particularly our LUT.

16. Project Timeline

17. Plans for Optimization • Reformulate problem slightly to avoid expensive sqrt() operation (requires a new LUT) • Ensure that LUT is mapping to ROM and buffer is on-chip RAM, not I/O pins • Determine required bit-width for math operation result and use minimum • Zero-pad arrays to be a power of two (avoids extra MUX elements) • Avoid division- add lookup for radius values • Experiment with size of input blocks. rgb2yuv example seemed to indicate large blocks create low latency, high area designs. • Choose pipelining interval, loop to unroll. Unroll low level, pipeline top?

18. Meeting with Mike Bradley • He showed us how to ensure that LUT is mapping to ROM in Catapult C GUI • Determined which for loops to unroll and which to pipeline • Discussed different types of technologies ASIC vs FPGAs

19. Updated Catapult Synthesis -Catapult C no longer crashes for large input images -Significant improvement in area, latency, and throughput -Processes 1 pixel per clock cycle

20. Questions? ?