MRF-Based Edge-Directed Interpolation

1. MRF-Based Edge-Directed Interpolation Min Li and Truong Nguyen Digital Video Processing lab., http://videoprocessing.ucsd.edu ECE Dept., University of California, San Diego

2. Outline • The spatial interpolation problem: applications and challenges • MRF-based edge-directed interpolation method • Concepts in MRF models • Two-dimensional Discontinuity-Adaptive Smoothness (DAS) constraint • The proposed MRF model • Applications in spatial interpolation • Simulation results • Conclusions and future work

3. Application Scenario The Spatial Interpolation Problem Definition

4. Flow Diagram MRF model Formulation of the 2-D DAS constraint U(ω) MAP The application in spatial interpolation Implementation & Simulation results

5. MRF Model In MRF model, an image is regarded as a 2-D random field on a 2-D lattice. Furthermore, in this random field, there are [Stan Z. Li, 2001] S.Z. Li, Markov Random Field Modeling in Image Analysis, Springer-Verlag,2001. [Geman&Geman, 1984] S. Geman and D. Geman,``Stochastic Relaxation, Gibbs distribution, and the Bayesian restoration of images”,vol.6,no.6,pp. 721—741,Nov. 1984.

6. Concepts in an MRF Model Cliques Neighborhood structure Potential and energy functions:

7. MAP-MRF Formulation The interpolation problem can be stated as the interpolation of the high resolution image given the available low resolution data. The Maximum A Posteriori (MAP) solution corresponds to the minimal energy state of the high resolution image if it is modeled as an MRF.

8. 1-DDAS Discontinuity features are related to large intensity variations, of which the potentials are bounded. [Stan Z. Li,1995] S. Z. Li,``On discontinuity-adaptive smoothness priors in computer vision”, IEEE Trans. on Pattern and Machine Intelligencevol.17,no.6, pp.576—586,June 1995.

9. Formulation of 2-D DAS Constraint : bounded energy in each direction where : direction weights, between 0 and 1

10. 2-D DAS: Direction Weights Calculation

11. An Example of Direction Weights magnitude Weights of the central Pixel is calculated row col. Edge pixel Weights in sixteen discrete directions

12. Implementation A Interpolation initialization (bilinear, spline, etc.) B Candidate set propose Pixel from low res. image 7x7 local window (16 discrete directions) Pixel to be interpolated Example pixel D Iteratively, I. New candidate propose II. Local energy change calculation. III. Updating the interpolation results based on energy difference. C Local energy calculation I. under 2-D DAS constraint. II. Using direction weights.

13. Implementation: Monte Carlo Markov Chain search • Two configurations, ω1 and ω2, are the same except for a single pixel (i, j). • The global probability of each is p(ω1)=exp{-U(ω1)/T}/Z and p(ω2)=exp{-U(ω2)/T}/Z, then, p(ω1)/p(ω2)= exp {(U(ω1)-U(ω2))/T} = exp{ ΔU/T}. • The updating rule is to accept the new state with probability Pc=min(1, p(ω1)/p(ω2)). • Update the probability of the pixel candidates with Pc . To calculateΔU, ΔU=E1(i, j)-E2(i, j)

14. Implementation 10 iterations initial state 4 iterations 20 iterations 40 iterations original

15. Interpolation Results Original Proposed, PSNR: 34.1dB NEDI, PSNR: 33.8dB Bicubic, PSNR: 32.5dB

16. Interpolation Results :Zoom-in Comparison Original Bicubic NEDI Proposed

17. Conclusions and Future Work Conclusions: • MRF-based edge-directed image interpolation. • MRF-MAP formulation of the spatial interpolation problem. • Formulation of the 2D DAS constraint. • Imposing the 2D DAS constraint to images via MRF model • Applications in spatial interpolation. • Obtaining sharp and consistent reconstructed edges Future Work: • Other applications in content-adaptive post-processing. • Stronger local content-adaptive property.

18. End. Author’s email: m3li@ucsd.edu

19. Single Pass Implementation • Iterations in optimization are removed. • Major complexity is with the learning of the model parameters. • Sliding window method. • In addition, two other options to lower the complexity. • Size-limited candidate set. • Discrimination of edge and non-edge pixels. [Freeman, 2002] W. T. Freeman, et al., ``Example-based super-resolution”, IEEE Transaction on Computer Graphics and Application, vol. 22, no. 2, pp. 56—65, Apr., 2002.

20. Traditional Interpolation Methods • Polynomial-based interpolation methods such as bilinear, bicubic and spline. • Polynomial-based interpolation followed by edge sharpening or enhancements. • Edge detection followed by edge-directed interpolation. • Edge-directed interpolation based on local correlation formulation [Li, 2001]. Challenges: • It’s challenging to interpolate sharp and consistent edges. • It’s difficult to detect natural edges (position, thickness, etc). [Li, 2001] X. Li and M. T. Orchard, ``New edge-directed interpolation”, IEEE Trans. on Image Processing, vol.10,no.10, pp. 1521—1526,2001.

21. Ideal and Natural Edges and Object Boundaries Natural edges Explicit edge detectors: Ideal step edges  works with ideal step edges.  has difficulties with natural edges. (e.g. edge position and thickness, corners and crossings)

22. Direction Weights

23. Implementation: Model Parameters

24. 2-D DAS: Direction Weights Calculation Choose window W Calculate PIVs Derive weights Flow diagram Take one direction (k, q) as an example to show the calculation 1) Window W has adaptive size 2) PIVs are calculated according to 3) Weights are derived as , or

25. Implementation: Model Parameters T can be constant (in Metropolis method [Metropolis, 1953]) or gradually decreases (in Simulated Annealing method [Geman&Geman, 1984]). One updating equation of T can be [Metropolis, 1953] N. Metropolis, et al., ``Equation of state calculations by fast computing machines, J. Chem. Phys.,vol. 21, pp. 1087—1092, 1953.