Sparse Representation for Image Reconstruction, and Face Recognition?

1. Sparse Representation for Image Reconstruction, and Face Recognition? Lei Zhang Dept. of Computing The Hong Kong Polytechnic University http://www.comp.polyu.edu.hk/~cslzhang

2. My Recent Research Focus • Sparse Representation and Low Rank • Image Reconstruction/Restoration • Pattern Recognition • Image Segmentation and Segmentation Evaluation • Region Merging, Graph Cut, ACM, Level Set • Image Quality Assessment • Our FSIM index achieves the best FR-IQA results so far • Visual Tracking • We recently developed a very fast but robust tracker • Biometrics • Finger Knuckle Print, Palmprint, Face…

3. Sparse representation • Image Restoration • Adaptive sparse domain selection • Centralized sparse representation • Face recognition • Robust (sparse) coding • Is it the sparsity that helps face recognition?

4. Image restoration Degradation model:

5. Image restoration • Inverse problem, regularized solution: • Conventional regularizations: TV and its variants … • Sparse modeling based regularization 5

6. Sparse Representation Modeling • Sparse model: • Sparsity-based regularization • Non-convex: • Convex: =  • Dictionary learning • K-SVD [TSP06]

7. Sparsity-based Image Restoration • Local Sparsity-based regularization • The drawback of universal dictionary • Potentially unstable • Too many atoms irrelevant to the given patch, low correlation

8. Adaptive dictionary design • Motivation • Locally adapting the atoms to signal • Orthogonal dictionary increases the stabilities of approximation • Idea • Design a set of compact dictionaries • Assign a sub-dictionary to a local patch, according to its content. W. Dong, L. Zhang, G. Shi and X. Wu, “Image deblurring and supper-resolution by adaptive sparse domain selection and adaptive regularization,” TIP 2011. Code available at: http://www4.comp.polyu.edu.hk/~cslzhang/ASDS_AReg.htm 8

9. Training set construction • Training dataset construction • Clustering dataset into K clusters • Feature extraction by high-pass filter • K-mean algorithm 9

10. A clustering example Image patches Image patches clustered into 5 clusters 10

11. The centroids The clustering centroid images of a training dataset 11

12. Sub-dictionaries learning • Learn a dictionary for each cluster • Learn an overcomplete dictionary by K-SVD? No • Computationally heavy; too many atoms; • The samples in a cluster have similar patterns • Learn a compact dictionary by PCA • Succeed in image denoising[Zhang, TIP09, PR10] 12

13. Example of learned sub-dictionaries Left column: the centroids of clusters; the right eight columns show the first eight atoms in the learned sub-dictionaries from the corresponding sub-datasets. 13

14. Adaptive sub-dictionary selection • Select a sub-dictionary according to the local contents • Obtain an initial estimate of xidenoted by • Nearest neighbor search • Update the estimate of xi • Other selection criterion: correlation based 14

15. Spatially adaptive regularization • Two other priors • Local stationary process, well characterized by Autoregressive (AR) model • Nonlocal self-similarity constraint • Spatially adaptive regularizations • AR model based regularization: • Nonlocal mean based regularization: 15

16. The proposed image restoration algorithm http://www4.comp.polyu.edu.hk/~cslzhang/ASDS_AReg.htm • The overall objective function: 16

17. The iterative shrinkage algorithm 17

18. Experiments: deblurring 18

19. Experiments: deblurring Original Noisy & blurred ASDS-AR-NL ASDS ASDS-AR 19

20. Experiments: deblurring Original Noisy & blurred ASDS-AR-NL Surrogate TV [TIP09] 20

21. Experiments: deblurring Original Noisy & blurred Surrogate TV [TIP09] ASDS-AR-NL 21

22. Experiments: super-resolution 22

23. Experiments: super-resolution Original LR ASDS ASDS-AR-NL ASDS-AR 23

24. Experiments: super-resolution Original LR Surrogate Softcut [TIP09] Yang et al. [CVPR08] ASDS-AR-NL 24

25. Experiments: super-resolution Original LR Surrogate Softcut [TIP09] Yang et al. [CVPR08] ASDS-AR-NL 25

26. Related publications • W. Dong, L. Zhang, G. Shi and X. Wu, “Image deblurring and supper-resolution by adaptive sparse domain selection and adaptive regularization,” TIP 2011. Matlab code available at: http://www4.comp.polyu.edu.hk/~cslzhang/ASDS_AReg.htm • W. Dong, G. Shi, L. Zhang and X. Wu, “Super-resolution with nonlocal regularized sparse representation,” in SPIE VCIP 2010. (Best paper award) • Lei Zhang, Weisheng Dong, D. Zhang and G. Shi, “Two-stage Image Denoising by Principal Component Analysis with Local Pixel Grouping,” Pattern Recognition, vol. 43, issue 4, pp. 1531-1549, April 2010. http://www4.comp.polyu.edu.hk/~cslzhang/LPG-PCA-denoising.htm • Lei Zhang, R. Lukac, X. Wu and D. Zhang, “PCA-based Spatially Adaptive Denoising of CFA Images for Single-Sensor Digital Cameras,” IEEE Trans. on Image Processing, vol. 18, no. 4, pp. 797-812, April 2009. http://www4.comp.polyu.edu.hk/~cslzhang/dmdn.htm • Lei Zhang, Weisheng Dong, Xiaolin Wu, Guangming Shi, “Spatial-Temporal Color Video Reconstruction from Noisy CFA Sequence,” IEEE Trans. on Circuits and Systems for Video Technology, 2010.

27. Centralized Sparse Representation • W. Dong, L. Zhang and G. Shi, “Centralized Sparse Representation for Image Restoration”, in ICCV 2011. Code available at: http://www4.comp.polyu.edu.hk/~cslzhang/code/CSR_IR.zip • A simple but very effective model was proposed. • It outperforms most of the state-of-the-arts in denoising, deblurring and super-resolution!

28. The key idea • For true signal • For degraded signal • The sparse coding noise (SCN) • To better reconstruct the signal, we need to reduce the SCN because: = yx

29. Centralized Sparse Representation • The proposed objective function • Key idea: Suppressing the SCN • How to estimate x? • The unbiased estimate: • The zero-mean property of SCN makes

30. Nonlocal redundancy • Exploiting nonlocal self-similarity • Nonlocal Mean Filtering [CVPR05] • BM3D [TIP07] • Improving the robustness of the sparse coding • Simultaneously Sparse Coding [ICCV09]

31. Centralized Sparse Representation • The nonlocal estimation of • The simplified objective function • The iterative solution:

32. Distribution of SCN: Laplacian The distribution of SCN for Lena image (a) noisy and blurred; and (b) is down-sampled. (c) and (d) show the same distributions in log domain.

33. ,  and p • The determination of  and  • Let , the MAP estimator •  and  are found nearly uncorrelated, and can be modeled by i.i.d Laplacian distribution. Thus Gaussian likelihood term

34. ,  and p • The MAP estimator leads to • The selection of dictionary • Locally learned: K-means clustering + PCA W. Dong, L. Zhang, et al [TIP11], [VCIP10]

35. Overall Procedure of CSR

36. Deblurring results Blur kernel: 9-by-9 uniform blur kernel; Gaussian noise of std dev. Table 1 The PSNR comparison with current state-of-the-arts methods

37. Deblurring Visual Comparison Blurred FISTA (27.75 dB) BM3D (28.61 dB) CSR (30.30 dB) Fergus, et al [SIGGRAPH06] Blurred Close up View CSR

38. Image super-resolution results Experiment condition: downsampling factor 3; 7x7 Gaussian with std dev 1.6; Gaussian noise with std dev 5. Table 2 The PSNR comparison results

39. Image super-resolution results LR TV (31.24 dB) Sparse (32.87 dB) CSR (33.68 dB) LR TV (31.34 dB) Sparse (31.55 dB) CSR (34.00 dB)

40. Sparse representation • Image Restoration • Adaptive sparse domain selection • Centralized sparse representation • Face recognition • Robust (sparse) coding • Is it the sparsity that helps face recognition?

41. y D α A testing sample sparse representation coefficient An overcomplete dictionary (Training samples) Sparse Representation based Classification (SRC) John Wright, Allen Yang, Arvind Ganesh, Shankar Sastry, and Yi Ma. Robust Face Recognition via Sparse Representation. IEEE Trans. Pattern Anal. Mach. Intell., 31(2): 210 – 227, 2009.

42. y D De α A testing sample sparse representation coefficient An overcomplete dictionary (Training samples and occlusion dictionary) β SRC with occlusion If there is corruption or occlusion, an additional dictionary is needed.

43. Some problems of SRC • Big-size dictionary, especially when there is the occlusion dictionary. • Holistic features. The claim made by Wright et al. that feature extraction is not so important to SRC actually holds only for holistic features. • To solve the above two problems, we proposed the Gabor based SRC (GSRC).

44. GSRC (ECCV 2010)M. Yang and L. Zhang, “Gabor Feature based Sparse Representation for Face Recognition with Gabor Occlusion Dictionary,” ECCV 2010. http://www4.comp.polyu.edu.hk/~cslzhang/code/ECCV10/GSRC_ECCV.rar Instead of using the original face image, we utilize the Gabor features of face images in SRC. Gabor Transform Uniform Down-sampling

45. GSRC • Based on sparsity constraint, an algorithm of Gabor occlusion dictionary learning is proposed. • GSRC’s important merit is the occlusion dictionary of GSRC can be compressed (often with a ratio 40:1~50:1), with higher recognition rate.

46. GSRC result Recognition rates by SRC and GSRC versus feature dimensionon (a) Extended YaleB and (b) AR database

47. GSRC results FERET pose subset

48. GSRC result

49. Fisher Discrimination Dictionary Learning for Sparse Representation M. Yang, L. Zhang, X. Feng and D. Zhang, “Fisher Discrimination Dictionary Learning for Sparse Representation,” in ICCV 2011. Code available at: http://www4.comp.polyu.edu.hk/~cslzhang/code/FDDL.zip

50. Motivation Predefined dictionary: e.g., all the training samples • Drawbacks of predefined dictionary: • Not effective enough to represent the query images due to the uncertain and noisy information in the original training images • The number of atoms of such a dictionary can be big • Not fully exploit the discriminative information hidden in the training samples.