Image Processing on the GPU

1. Image Processingon the GPU Edge Detection (with implementation on a GPU) And Text Recognition (if time permits) Jared Barnes Chris Jackson

2. Edge Detection • Wikipedia: Identifying points in a digital image at which the image has discontinuities.

3. Edge Detection http://4.bp.blogspot.com/-p_9w91wC_Rc/TbBgF7dQYhI/AAAAAAAAACM/DQTrM_a7Apg/s1600/edge-example-c.png

4. Canny Edge Detector • John Canny • “A Computational Approach to Edge Detection” • 1986 http://ieeexplore.ieee.org.libproxy.mst.edu/stamp/stamp.jsp?tp=&arnumber=4767851

5. Canny Edge Detector • Noise Removal • Image Gradient Computation • Non-Maximum Suppression • Hysteresis Thresholding

6. Noise Removal • Gaussian Smoothing or Blurring • A pixel is changed based on a weighted average of itself and its neighbors • The number of neighbors (3x3, 5x5) and the relative weights can vary 3D Gaussian Distribution Normalized 2D Gaussian Approximation http://homepage.cs.uiowa.edu/~cwyman/classes/spring08-22C251/homework/canny.pdf http://www.librow.com/content/common/images/articles/article-9/2d_distribution.gif

7. Gaussian Smoothing – Avoid Blur Too much Smooth Spotty About right http://www.eversparkinteractive.com/wp-content/uploads/2013/03/gaussian-blur-thumbnail.jpg http://media.tumblr.com/ccd6945141b46e5e2f5c36168f6a8037/tumblr_inline_mhcv1l0EZB1qz4rgp.png

8. Gradient Computation Sobel Operator (2 kernels) • Calculus! • First derivative in the X and Y directions (separately) Gx Gy Then round to: 0° =←→ 90°=↑↓ 45°=↗↙ 135°=↘↖ http://homepage.cs.uiowa.edu/~cwyman/classes/spring08-22C251/homework/canny.pdf

9. Image Gradient: Sobel Operator (Vertical Edges) Y Gradient X Gradient (Horizontal Edges) http://suraj.lums.edu.pk/~cs436a02/CannyImplementation.htm

10. Non-Maximum Suppression • Make edges exactly one pixel thick • Look at the gradient magnitude of your 2 neighbors in the direction of your angle Example 1 Angle = 135° ↘↖ Keep it! Example 2 Angle = 0° ←→ Kill it!

11. Non-Maximum Suppression Thick Edges Thin Edges (Gradient Magnitude) (Gradient Magnitude) http://suraj.lums.edu.pk/~cs436a02/CannyImplementation.htm

12. Hysteresis Thresholding Thigh = 45 Tlow = 35 • Two thresholds are better than one! • If a pixel’s value is above Thigh, it’s an edge. • If a pixel’s value is below Tlow, it’s not an edge. • If a pixel’s value is between Thigh and Tlow, it might be an edge (provided it is connected to an actual edge)

13. Canny Edge Detector • Smooth image to reduce noise • Calculate X & Y derivatives to get edges • Thin all edge widths to 1 pixel • Remove weak, unconnected edges (ta da!)

14. Parallelization! • How do we parallelize the Canny Edge Detector?

15. Parallelize: Canny Edge Detector • Convolution – Independent of order Image Kernel 10 5 2 2 5 2 10 2 2 2 10 5 2 2 10 20 4 10 40 4 10 20 2 2 2 2 20 40 2 2 20 40 40 20 2 2 Element-wise Multiplication Sum All Values Divide by Kernel Sum 11 230 11

16. Step 1: Gaussian Smoothing • Convolve a Gaussian Kernel with the image • Each GPU core can convolve each pixel in the image individually with the Gaussian Kernel • One thread per pixel, each performing 9 multiplies, 9 adds, and 1 division • Embarrassingly Parallelwith huge speedup

17. Step 2: Gradient Computation • Convolve two Sobel Kernels with the image • Wait, convolution again? • Same as previous step – we can even reuse the convolution function!

18. Step 3: Non-Maximum Suppression • Comparing 3 pixel gradient magnitudes and clearing the middle pixel or leaving it alone • Similar to convolution… but simpler! • Each GPU thread owns a pixel: • Check gradient angle of pixel • Compare this pixel’s magnitude with two neighbors in the direction of its angle • If I’m greater than those neighbors, leave me alone; otherwise, mark me as “not an edge” • Less speedup than steps 1 and 2

19. Step 4: Hysteresis Thresholding • Mark pixels > Thigh as strong edges • Mark pixels < Tlowas not edges • Mark remaining pixels as weak edges if they connect to a strong edge • Typically implemented with recursion • Each thread with a weak-edge pixel looks at nearest 2 neighbors to find a strong-edge pixel • With identical algorithms on CPU and GPU, speedup is marginal (memory accesses, not much processing)

20. Optical Character Recognition • Wikipedia: The mechanical or electronic conversion of images of printed text into computer-readable text. http://hackadaycom.files.wordpress.com/2010/09/helloworldconsole.png http://www.flacom.com/content/uploads/2013/09/hello-world.jpg

21. I have the edges – Now what? • Label Connected Components • Look For Letters • Adjust for disconnected letters HELLO WORLD HELLOWORLD E F ? ü j i

22. Connected Components Labelling • Create a list of components in the image • A component is simply a set of connected edges • Label each edge pixel with a unique component ID • Examine each pixel’s 8 touching neighbors and set that pixel’s ID to the smallest neighbor ID • Repeat step 2 until no pixel IDs are changed

23. Look For Letters • Uhh… what’s a letter? • How do we know it’s a letter? • How does the computer know it’s a letter?

24. Wait, what is a letter? • Letters are represented by a vector of numbers indicating the ratio of black pixels to white pixels in each division of the letter-image. A 0 0 5 0 15 60 15 0 10 55 40 0 15 75 40 55 10 60 15 0 5 15 0 0 0

25. Okay – Now look for letters • Compute how closely each labelled component matches each letter in your alphabet • The component is then marked with whichever letter it most closely matches

26. Correct for Disjoint Letters • Letters like ‘i’ and ‘j’ have floating parts • Sometimes edge detection may accidentally break up a letter • A letter vector should then get an additional property indicating vertical discontinuity L E T T E R V E C T O R … … … 0/1