Faculty of Computational Mathematics and Cybernetics, Moscow State University

1. Applications of Projection Method of Fourier Transform Inversion in Multimedia Data Treatment Faculty of Computational Mathematics and Cybernetics, Moscow State University Andrey S. Krylov NUS, Singapore, August 14, 2003

2. Applications of Projection Method of Fourier Transform Inversion in Multimedia Data Treatment • Outline: • Projection Method (Hermite series approach) • Applications • Image filtering and deblocking by projection filtering • Image matching • Texture matching • Low-level methods for audio signal processing

3. Hermite transform The proposed methods is based on the features of Hermite functions. An expansion of signal information into a series of these functions enables one to perform information analysis of the signal and its Fourier transform at the same time.

4. A) B) They derivate a full orthonormal in system of functions. The Hermite functions are defined as:

5. 2D decoded image by 45 Hermite functions at the first pass and 30 Hermite functions at the second pass Original image Difference image (+50% intensity)

6. 2D decoded image by 90 Hermite functions at the first pass and 60 Hermite functions at the second pass Original image Difference image (+50% intensity)

7. Image filtering and deblocking by projection filtering

8. Original lossy JPEG image

9. Enhanced image

10. Difference image (Subtracted high frequency information)

11. Zoomed in: Original image Enhanced image

12. Scanned image Enhanced image

13. Zoomed in: Enhanced image Scanned image

14. Image matching

15. Information parameterization for image database retrieval Algorithm Image normalizing Graphical information parameterization Parameterized image retrieval _

16. Information parameterization for image database retrieval = + HF component LF component Normalized image Recovered image with the recovered image plane Algorithm Image normalizing Graphical information parameterization Parameterized image retrieval

17. Information parameterization for image database retrieval Database size Initial images format Normalized images format Number of parameterization coefficients Error rate Search time – 768 images (4.12Gb) – 1600x1200x24bit (5.5Mb) – 512x512x24bit (0.75Mb) – 32x32x3 – <0.14% – 4sec. (forK7-750) Algorithm Image normalizing Graphical information parameterization Parameterized image retrieval

19. Image matching results

20. Image matching results

21. Texture matching

23. A method of obtaining the texture feature vectors Input function Fourier coefficients 1-D to 2-D expansion

24. A method of obtaining the texture feature vectors 1-D to 2-D expanded Hermite functions: 1-D Hermite functions:

25. Orientations

26. Localization problem Decomposition process is optimal, iflocalization segmentsof the input function and filtering functions are equal.

27. Feature vectors Standard coding In this approach to get the feature vectors we consider the functions n(x,y)where n1=0..64, and 6 energy coefficients are calculated as: E1 = (0)2 + (1)2 , E2 = (2)2+(3)2 + (4)2, E3= (5)2 +(6)2+(7)2+(8)2, … E6 = (33)2+(34)2 + … + (63)2 +(64)2 , f(x,y) is the source image.

28. Feature vectors Hierarchical coding E1 = (0(1))2 + (1(1))2 , E2 = (0(2))2+(1(2))2 + (2(2))2 +(3(2))2 , … E6 = (0(6))2+(1(6))2 + … + (62(6))2 +(63(6))2 , f(x,y) is the source image.

29. Feature vectors Hierarchical coding without subtractions E1 = (0(1))2 + (1(1))2 , E2 = (0(2))2+(1(2))2 + (2(2))2 +(3(2))2 , … E6 = (0(6))2+(1(6))2 + … + (62(6))2 +(63(6))2 , f(x,y) is the source image.

30. Feature vectors

31. a b Standard coding Hierarchical coding Hierarchical coding without subtractions a1 a2 0.004505 0.005349 0.003739 b1 b2 0.009905 0.008160 0.007742 c1 c2 0.017778 0.011848 0.009946 a3 a4 0.008807 0.006047 0.002422 b3 b4 0.020075 0.010667 0.006275 c3 c4 0.128322 0.101591 0.080176 a1 b1 0.488420 0.492238 0.491653 b1 c1 0.315644 0.304597 0.305442 Feature vectors

34. Image segmentation task:Brodatz textures

35. Image segmentation task

36. Image segmentation task

37. Image segmentation task

38. Texture parameterization using 2-D Hermite functions

39. Low-level methods for audio signal processing Quasiperiod’s waveforms N O L

40. Areas of Hermite transform application: • Signal filtering • Speaker indexing • Speaker recognition using database • Source separation

41. Audio sample Music Speaker 1 Speaker 2 2 speakers

42. Quasiperiod waveform Hermite histogram N O L

43. Speaker indexing

44. Speech/music separation Currently implemented criterions: • Analysis of signal dynamics (distribution of amplitudes in time) • Analysis of base tone distribution in time Result of speech/music separation:

45. Mix detection One speaker Mix

46. Applications of Projection Method of Fourier Transform Inversion in Multimedia Data Treatment • References • Krylov A.S., Liakishev A.N.,Numerical Projection Method For Inverse Fourier Transform and its Application,Numerical Functional Analysis and optimization, vol. 21 (2000) p. 205-216. • Krylov A.S., Kortchagine D.N.,Projection filtering in image processing, Graphicon’2000 Conf. proceedings, p. 42-45, 2000. • Krylov A.S., Kortchagine D.N., Lukin A.S., Streaming Waveform Data Processing by Hermite Expansion for Text-Independent Speaker Indexing from Continuous Speech, Graphicon’2002 Conf. proceedings, p. 91-98, 2002. • Krylov A.S., Kutovoi A.V., Wee Kheng Leow, Texture Parameterization With Hermite Functions,Graphicon’2002 Conf. proceedings, p. 190-194, 2002. • Mohsen Najafi, Andrey Krylov, Danil Kortchagine Image Deblocking With 2-D Hermite Transform, Graphicon’2003 Conf. proceedings.