Correntropy as a similarity measure

1. Correntropy as a similarity measure Computational NeuroEngineering Laboratory University of Florida http://www.cnel.ufl.edu weifeng@cnel.ufl.edu Acknowledgment: This work was partially supported by NSF grant ECS-0300340 and ECS-0601271. Weifeng Liu, P. P. Pokharel, Jose Principe

2. Outline • What is correntropy • Interpretation as a similarity measure • Correntropy Induced Metric robustness • Applications

3. Correntropy: General Definition For random variables X, Y correntropy is where K is Gaussian kernel Sample estimator

4. Correntropy = ‘Correlation’ + ‘Entropy’ • Correlation with high order moments • Taylor expansion of Gaussian kernel • Kernel size large, second order moment dominates • Average over dimensions is the argument of Renyi’s quadratic entropy

5. Reproducing Kernel Hilbert Space induced by Correntropy- (VRKHS) • V(t,s) is symmetric and positive-definite • Defines a unique Reproducing Kernel Hilbert Space---VRKHS • Wiener filter is an optimal projection in RKHS defined by autocorrelation • Analytical nonlinear Wiener filter framed as an optimal projection in VRKHS

6. Probabilistic Interpretation • Integration of joint PDF along x=y line • Probability of • Probability density of X=Y

7. Probabilistic Interpretation

8. Geometric meaning • Two vectors • Define a function CIM

9. Correntropy Induced Metric • CIM is Non-negative • CIM is Symmetric • CIM obeys the triangle inequality Therefore it is a metric that is induced in the input space when one operates with correntropy

10. Metric contours • Contours of CIM(X,0) in 2D sample space • close, like L2 norm • Intermediate, like L1 norm • far apart, saturates with large-value elements (direction sensitive)

11. CIM versus MSE as a cost function • Localized similarity measure

12. CIM is robust to outliers • measure similarity in a small interval; Do not care how different outside the interval • Resistant to outliers (in the sense of Huber’s M-estimation)

13. Application 1: Matched filter • S transmitted binary signal • N channel noise • Y received signal

14. Application 1: Matched filter • Sampled (1,-1) received signal • Linear matched filter • Correntropy matched filter

15. Application 1: Matched filter BER SNR (dB)

16. Application 2: Robust Regression • X input variable • f unknown function • N noise • Y observation

17. Application 2: Robust Regression • Maximum Correntropy Criterion (MCC) y=g(x) X

18. MCC is M- Estimation MCC  

19. Significance • Correntropy is a building block of • correntropy nonlinear Wiener filter • correntropy matched filter • correntropy nonlinear MACE filter • correntropy Principal Component Analysis • Renyi’s quadratic entropy • This understanding is crucial to explain the behavior of nonlinear algorithms and high-order statistics!

20. References • [1] I. Santamaria, P. P. Pokharel, J. C. Principe, “Generalized correlation function: definition, properties and application to blind equalization,” IEEE Trans. Signal Processing, vol 54, no 6, pp 2187- 2186 • [2] P. P. Pokharel, J. Xu, D. Erdogmus, J. C. Principe, “A closed form solution for a nonlinear Wiener filter”, ICASSP2006 • [3] Weifeng Liu, P. P. Pokharel, J. C. Principe, “Correntropy: Properties and Applications in Non-Gaussian Signal Processing”, submitted to IEEE Trans. Signal Proc.