EE645: Independent Component Analysis

1. EE645: Independent Component Analysis Elliot Taniguchi Advisor: Dr. Kuh May 16, 2003

3. Motivation for ICA Cocktail Party Problem Suppose you are in a crowded room with many people. How do you understand what any one person is saying? Separation of Independent Signals Similar to Blind Source Separation Little knowledge of the signals Access to mixed signals only

4. Cocktail Party Problem

5. Cocktail Party Problem ICA Separation Algorithm Separation of Speech Signals Humans can separate multiple signals with only two ears/sensors ICA needs as many ears/sensors as message signals Here we assume he has four ears!

6. Recovered Messages

7. Mathematical Formulation Overview ICA Definition ICA Assumptions Independent Signals Non-Gaussian ICA Limitations Scaling Permutations No. of Sensors

8. ICA Definition Mixed Signals in Matrix Notation

9. ICA Solution Signal Separation Find using the ICA Algorithm

10. ICA Block Diagram (2 Signals)

11. ICA Assumption #1: Independence Probability Density Definition Expected Value Definition

12. ICA Assumption #2: Non-Gaussian Property of Gaussian signals Addition of two independent Gaussian random variables is another single Gaussian random variable. Information Lost! Kurtosis Function Special Case: kurt(N) = 0

13. Limitation #1: Scaling ICA maximizes independence between signals.

14. Limitation #2: Signal Permutations The mixing matrix and independent components are unknown.

15. Limitation #3: No. of Sensors Sensor Requirement The number of separated signals cannot be larger than the number of inputs. Current research is being done to reduce this constraint.

16. ICA Separation Technique Central Limit Theorem If two random (non-Gaussian) signals added, the resulting signal will be more Gaussian than the original two random signals ICA Separation Concept Central Limit Theorem (in Reverse) Maximizing Non-Gaussianity Results in separating the two signals

17. Fast ICA Algorithm Overview �Fixed-Point� Algorithm Implementation Fast ICA algorithm Extensions Algorithm Speed & Performance Currently the fastest Most Commonly Used

18. Fast ICA Algorithm Choose a random initial weight vector. Let, Let, Repeat until converges.

19. Fast ICA Extensions Preprocessing Normalize mean to zero Pre-Whitening Activation Functions g(u)=u^3 g(u)=u^2 g(u)=tanh(a1*u) g(u)=u*exp(-a2*u^2/2)

20. Noise Separation Example Separation of Noise Impulsive Noise Additive White Gaussian Noise Implementation Two Sensor Setup Fast ICA Algorithm

21. Noise Generation AWGN Gaussian R.V. Impulsive Noise Poisson R.V. Gaussian R.V.

22. Physical Setup

23. Noise Separation Example

24. Noise Separation Example

25. Noise Separation Example

26. Noise Separation Performance

27. Noise Separation Performance

28. Noise Separation & Feature Extraction ICA performs well in Blind Source Separation ICA for Feature Extraction Reduce Complexity of the Neural Network Train only on the appropriate signal Detection and Estimation of Hidden Signals

29. Noise Detection and Estimation: Method #1

30. Noise Detection and Estimation: Method #2

31. Noise Detection and Estimation:Conclusion Additional Preprocessing Segmentation of Impulsive Noise (Time-Limited) Possible Inputs to the Neural Network Statistical Moments Signal Samples Possible Neural Networks Back Propagation SVM Radial Basis Functions Problem Need to train Neural Network in Matlab

32. Digital Watermarking of Music Motivation Popularity of Digital Storage Devices Reliable, Fast, Ease of duplication, etc. How to protect copyrighted information? Leaving digital signatures of its artist Essential Properties for Watermarking Undetectable Irremovable Resilient

33. Detection & Estimation of Watermarks Detection of Watermark Authenticate copyrighted music Estimation of Watermark Authenticate copyrighted music Information on artist, producer, etc.

34. Watermarking Model Process Mix the original musical data with watermark Keep watermark Power relatively low Ensure high quality of the watermarked music Watermark is better hidden

35. Popular Digital Formats

36. Detection of Watermark Watermarking Detection Algorithm Use the ICA model to randomly mix the watermark and music file. Save the watermarked music in the popular *.wav format Read the saved *.wave file. Separate the watermark and the music file. Identify the watermark using statistical methods (mean, std, etc.) Performance Statistic Correlation Coefficient (Absolute Value)

37. Detection Performance

38. Estimation of Watermark Watermarking Estimation Algorithm Use the ICA model to randomly mix the watermark and music file. Save the watermarked music in the popular *.wav format Read the saved *.wave file. Separate the watermark and the music file. Identify the watermark using statistical methods (mean, std, etc.) Digitize the watermark signal. Performance Statistic Bit Error Rate

39. Estimation Performance

40. Is the Music Content Preserved?

41. Resilience of Proposed Watermark Resiliency Previous Simulations show that wav format is resilient to 8-bit and 16-bit quantization. Can the watermark be detected after Mp3 compression and decompression?

42. Mp3 Compression/Decompression Actual Mp3 compression program used CDex Version 1.40 Release Mp3 (lossy) Compression Down Sampling Filter banks Mp3 Decompression Up Sampling Reconstruction Filter

43. Detection and Estimation Performance Bad News Correlation Coefficients 0 MSE 0.5 Possible Problems with Mp3 Compression Down Sampling Watermark information is lost Quantization Noise Watermark information absorbs into the quantization noise Lossy Compression 11:1 Compression Rate

44. How to Improve its Resilience? Alternative Approaches Synchronization of the music data Time shift in Mp3 compression? Storing watermark in certain frequencies (where less quantization occurs) Error Coding Hamming Reed-Solomon

45. Conclusion Wave to Wave Format Very good performance (even for 8-bit wave files) SNR is very low. Music Integrity is excellent Mp3 Compression Very bad performance Alternative methods need to be found! Need a greater understanding of current Mp3 Compression Algorithms

46. References [1] Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003. [2] Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000. [3] Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis. http://www.cis.hut.fi/~aapo/. [4] Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. http://www.cis.hut.fi/~aapo/. April 1999. [5] Hyvarinen and Oja. Independent Component Analysis: A Tutorial. http://www.cis.hut.fi/projects/ica/. April 1999. [6] Introduction to Blind Source Separation. http://www.cnl.salk.edu/~tewon/Blind/. [7] Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003. [8] Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998. [9] Shen and others. A Method for Digital Image Watermarking Using ICA. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.