המעבדה למערכות ספרתיות מהירות

1. Technion - Israel institute of technology department of Electrical Engineering הטכניון - מכון טכנולוגי לישראלהפקולטה להנדסת חשמל High speed digital systems laboratory המעבדה למערכות ספרתיות מהירות Denoising video in real time 2007 winter Final presentation (part B) Performed by: Cohen Ido,Volokitina Irina Instructor: Rivkin Ina , Technion Almog Asaf , Intel

2. Agenda • Motivation and background. • Optional concepts and solutions (Why bi-lateral filter?). • Validation’s expectations • Results’ validation (and more). • Implementation in hardware (gidel and synplify). • Conclusions • Demonstration

3. Image noise – definition and more • The term noise usually refers to the high frequency random perturbations. • corresponds to visible grain or particles present in the image. • Generally caused by the electronic noise in the input device sensor and circuitry (e.g. scanner, digital camera).

4. Type of noise

5. The solution is DENOISING • Removing noise from data is often the first step in data analysis. • Denoising techniques should not only reduce the noise, but do so without blurring or changing the location of the edges.

6. Optional concepts and solutions • Diffused image. • Bilateral filter.

7. Bilateral Bilateral and diffusion filtering comparison Diffusion • Simple implementation. • Noniterative. • Local. • YUV. (CIE) • Complicating implementation. • Iterative. • Not local. • RGB. Conclusion: local and noniterative characterizations make us to choose in bilateral algorithm

8. Bilateral filter • Bilateral filtering smoothes images while preserving edges, by means of a nonlinear combination of nearby image values. • The bilateral filter can enforce the perceptual metric underlying the CIE-Lab color space, and smooth colors and preserve edges in a way that is tuned to human perception.

9. VALIDATION • Implement the algorithm with high level design tool MATLAB, and compare the denoising image and the original one.

10. Validation the solution’s correctness • The indication for “success” or “fail” is MSE. The destination is to reduce the MSE by 50%.

11. configuration denoising unit (bilateral) RGB RGB RGB RGB Noise (normal distribution) + Output File Input File (without noise) RGB MSE measurement RGB RGB Validation Process noisemaker MSE redaction percentage

12. Implementation and validation process Stages in the implementation • MSE tester. • Normal distribution noisemaker. • Denoising unit. • rgb2yuv function and opposite. • Adaptive LPF. • Fir select. • Adaptive HPF.

13. RGB YUV YUV RGB Implementations stages Denoising unit (bilateral) YUV RGB RGB

14. RGB YUV YUV RGB Implementations stages Denoising unit (bilateral) Adaptive LPF RGB RGB YUV YUV configuration

15. RGB YUV YUV RGB Implementations stages Denoising unit (bilateral) Adaptive LPF MUX RGB RGB YUV Fir select YUV configuration

16. Implementations stages Denoising unit (bilateral) Adaptive LPF MUX YCbCr RGB Fir select RGB RGB YCbCr YCbCr YCbCr RGB Adaptive HPF configuration

17. expectations • rgb2yuv and opposite – changing up to 10% in MSE. • LPF – reducing MSE by at least 35% • Adding fir select – reducing MSE by 5-10% • HPF – reducing 10% MSE. Destination: reduction the MSE by 50%

18. Results • rgb2yuv and opposite – changing up to 10% in MSE => less than 1%. • LPF – reducing MSE by at least 35% => 44% • Adding fir select – reducing MSE by 5-10% => 20% • HPF – reducing 10% MSE => add about 10% to MSE. Bottom line: reduction the MSE by 50% (and more) achieved

19. Original

20. noisy

21. filtered

22. final

23. Noise No Noise Goals of project • Implement denoise bilateral algorithm configuration Denoise bilateral algorithm Video out Video in =

24. System Block Diagram Denoise algorithm Bilateral filter YCbCr RGB RGB YCbCr Video out Video in Synplify implemented configuration GIDEL implemented

25. X X + Bilateral filter block Diagram GIDEL implemented configuration Controller HPF (3X3) M E M O R Y YCbCr LPF (3X3) YCbCr Image analysis & Fir select Synplify implemented

26. Memory block Diagram SYNPLIFY implemented M E M O R Y L[n-2] YCbCr YCbCr L[n] YCbCr L[n-1] YCbCr L[n]

27. + HPF/LPF block SYNPLIFY implemented HPF/LPF L[n-2] L[n-2] FIR L[n-1] L[n-1] L[n-1] FIR L[n] L[n] FIR

28. - Image analysis & Fir select block diagram SYNPLIFY implemented MAX last 3 clock L[n-2] L[n-1] >=< L[n] MIN last 3 clock

29. Controller block diagram SYNPLIFY implemented User definition Threshold & weight parameter LPF coefficients HPF coefficients

30. Noise No Noise System block diagram Denoising unit (bilateral) Adaptive LPF MUX YCbCr RGB Fir select RGB RGB YCbCr YCbCr YCbCr RGB HPF configuration GiDELPROCWizard

31. (a) – input domain; (b) – delay line block; (c) – RGB to YCbCr transform blocks – one for each line; (d) – delay sample blocks; (e) – min max block; (f) , (g) – synchronizing pipes; (h1) – LPF block; (h2) –HPF block; (k) – filter select domain; (l) – synchronizing domain; (m) – YCbCr to RGB block; (n) – output domain (p) – simplify tool button Implementation : System level top view

32. Flow stagesStage1 - break input stream to lines • Input stream divided to 3 lines data stream while line to line delay is 800 pixels (screen resolution) by 3 registers when each one delayed from others according to line its responsible for .

33. Flow stages (cont.)Stage 2 - RGB to YCbCr transform one for each line Conversion algorithm taken from : http://en.wikipedia.org/wiki/YCbCr RGB data is 24 bit pattern while each 8bit of it is a value for each one of 3 colors . e.i YCbCr from "digital 8-bit RGB“ Y = 16 + 1/256*(65.738*R + 129.057*G + 25.064*B) Cb = 128 + 1/256*(-37.945*R - 74.494*G + 112.439*B) Cr = 128 + 1/256*(112.439*R - 94.154*G - 18.285*B) The inverse transform equivalent to the above is: 8-bit RGB from YCbCr R = ( 298.082*Y + 408.583*Cr) / 256 - 222.921 G = ( 298.082*Y - 100.291*Cb - 208.120 * Cr) / 256 + 135.576 B = ( 298.082*Y + 516.412*Cb) / 256 - 276.836

34. (a) – input domain; (b),(e) – synchronizing domain; (c) – expand domain to expand 24 bit incoming pattern to three 8 bit patterns (see General description); (d) - Conversion domain (see general); (f) – output domain; Flow stages (cont.)Stage 2 - RGB to YCbCr transform one for each line

35. Flow stages (cont.)Stage 3 - get 3x3 matrix surrounding for each pixel X 3 =

36. (a) – In_y1 to In_y9 - data input domain; (b) – Cb and Cr components of input; (c) – Control signals inputs ( de_enable , vsync. hsync); (d) – Min, max evaluation domain ; (e) – In_y1 to In_y2 data transfer domain; (f) – Control signals synchronizing domain ; (g) – min and max values output; (k) – In_y1 to In_y2 data output; (h) – Cb and Cr components of output; (j) – Control signals inputs; Blue outlined vertical areas are synchronizing pipelines. Flow stages (cont.)Stage 4 – min/max evaluation

37. Flow stages (cont.)Stage 4(cont.)– min/max evaluation Max-min = Delta = Filter select indicator

38. Flow stages (cont.)Stage 5 - LPF

39. Flow stages (cont.)Stage 6 - HPF HPF bypass mechanism

40. Flow stages (cont.)Stage 7 - Filter select mechanism (a) – threshold input; (b) – comparator element; (c) – MUX; (d) –input from HPF_3x3 block ; (e) - input from LPF_3x3 block; (f) – filtered Y component of pixel All other element are for synchronizing .

41. (a) – input domain; (b) - Conversion domain (see General description); (c) – synchronizing domain; (d) – concat domain to concat the three 8 bit patterns bits to 24 bit outgoing pattern to (see General description); (e) – output domain; Flow stages (last )Stage 7 - YCbCr 2 RGB

42. Summary and conclusionsSummary • We have implemented denoising bilateral algorithm using GIDEL's hardware and High Level Design tools as SynplifyDSP and SynplifyPro. • The algorithm performs real-time adaptive filtering per pixel of video stream coming from the DVI input . • Project goals are fully achieved.  Conclusions • Not easy to implement on such data rates but - Can be done • Better results using HPF bypass -> Implemented HPF algorithm is not a best solution • Possible improvement : • other HPF algorithm or another filter selection algorithm (few threshold etc.)

43. Questions • Demonstration