Reconfigurable Processor for Energy-Scalable Computational Photography

1. In the Name of God,the Compassionate, the Merciful Reconfigurable Processor for Energy-Scalable Computational Photography Class presentation for the course Custom Implementation of DSP Systems based on the ISSCC 2013 paper [1]: M. R. Azarbad Department of Electrical and Computer Engineering, University of Tehran, Tehran, Iran R. Rithe, P. Raina, N. Ickes, S. V. Tenneti, and A. P. Chandrakasan All the materials are copy rights of their respective authors as listed in references

2. Outline • Introduction • Basic Concepts • This Work • Chip Features • Conclusions • References

3. Introduction • Computational photography enable capture and synthesis of images that could not be captured with a traditional camera [1] • Non-linear filtering techniques like bilateral filtering in computational photography [1] • These techniques have a wide range of applications, including High-Dynamic Range (HDR) imaging, Low-Light Enhanced (LLE) imaging and video enhancement [1] • The real-time processing necessitates fast hardware implementations [1]

4. Basic Concepts • Filtering in image processing • Function of the values of the input image in a small neighborhood [2] • Gaussian low-pass filtering computes a weighted average of pixel values in the neighborhood (the weights decrease with distance from the neighborhood center) [2] Fig. 1: The mechanism of spatial filtering [3]

5. Basic Concepts • Images typically vary slowly over space, so near pixels are likely to have similar values [2] • Appropriate to average them together [2] • The noise values that corrupt these nearby pixels are mutually less correlated than the signal values [2] • Noise is averaged away while signal is preserved [2]

6. Basic Concepts How can we prevent averaging across edges, while still averaging within smooth regions? • Iteration may raise issues of stability and, depending on the computational architecture, efficiency [2] • bilateral filtering [2] bilateral filtering • The idea is to do in the range of an image (vs. traditional filters domain) [2] • Range filters are nonlinear (weight dependency on image intensity or color) [2] • Combination of range and domain filtering [2]

7. Basic Concepts bilateral grid • The bilateral grid is a three dimensional array [3] • The first two dimensions (x, y) correspond to 2D position in the image plane and form the spatial domain [3] • The third dimension z corresponds to a reference range(typically, the range axis is image intensity) [3]

8. This Work • Implementation of bilateral filtering using a reconfigurable grid [1] • scheduling the filtering engine so that only two grid rows need to be stored at a time [1] • Reduces the storage requirement to 21.5kB (vs. 65MB for a 10Mpixel image) [1] • The implementation is flexible to allow varying grid sizes for energy/resolution scalable image processing [1]

9. This Work Grid Assignment Engine • Computes the bilateral grid [1] • The input image is partitioned into blocks of size σs×σs [1] Fig. 2: The Grid Assignment Engine [1]

10. This Work Convolution Engine • Convolves the grid intensities and weights with a 3×3×3 Gaussian kernel, which is equivalent to bilateral filtering in the image domain [6], and returns the normalized intensity [1] Fig. 3: The Concolution Engine [1] Fig. 4: The Interpolation Engine [1]

11. This Work Fig. 5: Outputs of HDR imaging, LLE imaging and Glare reduction[1]

12. This Work Fig. 6: Energy vs. performance trade-off for a 10Mpixel image and the frequency of operation for a range of voltages. Run times for different image sizes at 98MHz, 0.9V [1]

13. Chip Feature • The grid processing tasks are scheduled to minimize local storage requirements and memory traffic [1] Fig. 7: Die photo of the testchip [1]

14. Conclusions • bilateral filtering using a reconfigurable grid, which reduces the storage requirement to 21.5kB [1] • The implementation is flexible to allow varying grid sizes for energy/resolution scalable image processing [1] • A 10Mpixel image in 771ms with 17.8mW power consumption while operating at 98MHz, 0.9V [1]

15. References [1] R. Rithe, P. Raina, N. Ickes, S. V. Tenneti, and A. P. Chandrakasan, “Reconfigurable Processor for Energy-Scalable Computational Photography,” ISSCC Dig. Tech. Papers, 2013. [2] C. Tomasi, R. Manduchi, “Bilateral Filtering for Gray and Color Images,” IEEE International Conf. Computer Vision, pp. 839-846, 1998. [3] R.C. Gonzalez, and R.E. Woods, “ Digital image processing,” 2nd edition, Prentice-Hall, 2002. [4] F. Durand, J. Dorsey, “Fast Bilateral Filtering for the Display of High- Dynamic-Range Images,” ACM Trans. on Graphics, vol. 21, no. 3, pp. 257-266, 2002. [5] G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, K. Toyama, “Digital Photography with Flash and No-Flash Image Pairs,” ACM Trans. Graphics, vol. 23, no. 3, pp. 664-672, 2004. [6] J. Oh, G. Kim, J. Park, I. Hong, H.-J. Yoo, “A 320mW 342GOPS Real-Time Moving Object Recognition Processor for HD 720p Video Streams,” ISSCC Dig. Tech. Papers, pp. 220-221, 2012. [7] J. Chen, S. Paris, F. Durand, “Real Time Edge-Aware Image Processing With the Bilateral Grid,” ACM Trans. Graphics, vol. 26, no. 3, article 103, 2007. [8] http://en.wikipedia.org

16. Thank You!