Independent Component Analysis: The Fast ICA algorithm

1. Independent Component Analysis: The Fast ICA algorithm Jonathan Kam EE 645

2. Overview The Problem Definition of ICA Restrictions Ways to solve ICA NonGaussianity Mutual Information Maximum Likelihood Fast ICA algorithm Simulations Conclusion This is a brief overview of what I would like to talk about today. The first thing we will go over is the problem or why people started to research ICA. I will then go over some of the restrictions that are need in order for this algorithm to work. After that, I will talk about different ways in which ICA can be solved and a little about the pros and cons of using the different methods Then I will go into detail about the Fast ICA algorithm I will then go over some simulations Conclude on other ways in which ICA could be usedThis is a brief overview of what I would like to talk about today. The first thing we will go over is the problem or why people started to research ICA. I will then go over some of the restrictions that are need in order for this algorithm to work. After that, I will talk about different ways in which ICA can be solved and a little about the pros and cons of using the different methods Then I will go into detail about the Fast ICA algorithm I will then go over some simulations Conclude on other ways in which ICA could be used

3. The Problem Cocktail Problem Several Sources Several Sensors Ex: Humans hear mixed signal, but able to unmix signals and concentrate on a sole source Recover source signals given only mixtures No prior knowledge of sources or mixing matrix aka Blind Source Separation (BSS) The original problem stems from the cocktail problem. The cocktail problem is basically the scenario that you are at a party with several conversations going on around the room. Even though we hear these conversations as mixed signals in our ears, humans are able to unmix these signals into different components, so that we can concentrate on a sole conversation. The model of the problem is this. You are have several sources which you have no prior information about. You then have several sensors that pick up a linear mixture of these source signals. How can you recover the source signals given only the mixtures. This problem is also commonly known as Blind Source Separation or Blind Signal SeparationThe original problem stems from the cocktail problem. The cocktail problem is basically the scenario that you are at a party with several conversations going on around the room. Even though we hear these conversations as mixed signals in our ears, humans are able to unmix these signals into different components, so that we can concentrate on a sole conversation. The model of the problem is this. You are have several sources which you have no prior information about. You then have several sensors that pick up a linear mixture of these source signals. How can you recover the source signals given only the mixtures. This problem is also commonly known as Blind Source Separation or Blind Signal Separation

4. Assumptions Source signals are statistically independent Knowing the value of one of the components does not give any information about the others ICs have nongaussian distributions Initial distributions unknown At most one Gaussian source Recovered sources can be permutated and scaled In order to solve this problem using ICA there are several assumptions that have to be made in order to correctly determine the independent components. The first assumption is that all source signals are statistically independent. This means that knowing the value of one of the components does not give any information about the other components. In order to do so these ICS have to have some unknown nongaussian distributions. If we knew the distributions to the sources, then this problem could be siimplified and you wouldn�t need ICA to solve it This assumption is not totally correct because BSS problem can be solved if there is at most one Gaussian source. I will talk later about why there needs to only be at most only one gaussian source The last assumption from the ICA algorithm is that the recovered sources will not be equal to the original source. They will be some permutated and scaled form the original signalIn order to solve this problem using ICA there are several assumptions that have to be made in order to correctly determine the independent components. The first assumption is that all source signals are statistically independent. This means that knowing the value of one of the components does not give any information about the other components. In order to do so these ICS have to have some unknown nongaussian distributions. If we knew the distributions to the sources, then this problem could be siimplified and you wouldn�t need ICA to solve it This assumption is not totally correct because BSS problem can be solved if there is at most one Gaussian source. I will talk later about why there needs to only be at most only one gaussian source The last assumption from the ICA algorithm is that the recovered sources will not be equal to the original source. They will be some permutated and scaled form the original signal

5. Definition of ICA Observe N linear mixtures x1,�,xn of n independent components xj = aj1s1 + aj2s2 + � + ajnsn, for all j aj is the column of the mixing matrix A Assume each mixture xj and each IC sk is a random variable Time difference between mixes dropped Independent components are latent variables Cannot be directly observed Now I will go over the basic definition of ICA. First is the input. ICA is given N observed linear mixtures x1 through xn for n independent components. The aj seen in the equation is the column data in the mixing matrix. The sk is the independent component that we wish to find. It is assumed that each mixture xj and each IC sk is a random variable For simplicity of this model the time difference between captured mix signals have been dropped The problem with receiving only the mixtures is that it makes the independent component latent variables, meaning they cannot be directly observedNow I will go over the basic definition of ICA. First is the input. ICA is given N observed linear mixtures x1 through xn for n independent components. The aj seen in the equation is the column data in the mixing matrix. The sk is the independent component that we wish to find. It is assumed that each mixture xj and each IC sk is a random variable For simplicity of this model the time difference between captured mix signals have been dropped The problem with receiving only the mixtures is that it makes the independent component latent variables, meaning they cannot be directly observed

6. Definition of ICA ICA Mixture model: x=As A is mixing matrix; s is matrix of source signals Goal Find some matrix W, so that s = Wx W = inverse of A Now that we understand the model of our incoming mixtures, the ICA model can be defined as x = As Where A is some unknown mixing matrix, s is our source signals and x is our observed mixtures. With this model we can then see that the goal of ICA is to find some matrix W, so that s = Wx This means that the ideal solution to ICA would be to find W so that it is the inverse of the mixing matrix ANow that we understand the model of our incoming mixtures, the ICA model can be defined as x = As Where A is some unknown mixing matrix, s is our source signals and x is our observed mixtures. With this model we can then see that the goal of ICA is to find some matrix W, so that s = Wx This means that the ideal solution to ICA would be to find W so that it is the inverse of the mixing matrix A

7. Definition: Independence Two functions independent if E{h1(y1)h2(y2)} = E{h1(y1)} E{h2(y2)} If variables are independent, they are uncorrelated Uncorrelated variables Defined: E{y1y2} = E{y1} E{y2} = 0 Uncorrelation doesn�t equal independence Ex: (0,1),(0,-1),(1,0),(-1,0) E{y12y22} = 0 ? � = E{y12} E{y22} ICA has to prove independence The key to solving the ICA algorithm is to find independent components. In order to determine whether or not two signals are independent there joint expectation have to be equal to the product of their expectations. (Independence of two functions h1 and h2 is defined as the expectation of h1 and h2 are equal to the expectation of h1 times the expectation of h2) By definition of independence, if variables are independent, they are also uncorrelated. However even though variables maybe uncorrelated, it doesn�t mean that they are independent. This slide shows an example where uncorrleated variables don�t come out to independent variables. Making the mixtures uncorrelated can be easily done as a preprocessing step because having uncorrelated variables is a step towards have independent variables, but ICA has to eventually find independent variabls The key to solving the ICA algorithm is to find independent components. In order to determine whether or not two signals are independent there joint expectation have to be equal to the product of their expectations. (Independence of two functions h1 and h2 is defined as the expectation of h1 and h2 are equal to the expectation of h1 times the expectation of h2) By definition of independence, if variables are independent, they are also uncorrelated. However even though variables maybe uncorrelated, it doesn�t mean that they are independent. This slide shows an example where uncorrleated variables don�t come out to independent variables. Making the mixtures uncorrelated can be easily done as a preprocessing step because having uncorrelated variables is a step towards have independent variables, but ICA has to eventually find independent variabls

8. ICA restrictions Cannot determine variances s and A are unknown Scalar multipliers on s could be canceled out by a divisor on A Multiplier could even be -1 Cannot determine order Order of terms can changed. With this proposed ICA model there are some restrictions what the ICA algorithm can solve. The first restriction is that ICA cannot determine variances. Since s and A are unknown any scalar multiplier on s could be canceled out by a divisor on A The output signal could even be scaled by a -1 which would reverse the amplitude of the signal The next restriction I that with this ICA model, there is no order to which the independent components are foundWith this proposed ICA model there are some restrictions what the ICA algorithm can solve. The first restriction is that ICA cannot determine variances. Since s and A are unknown any scalar multiplier on s could be canceled out by a divisor on A The output signal could even be scaled by a -1 which would reverse the amplitude of the signal The next restriction I that with this ICA model, there is no order to which the independent components are found

9. ICA restrictions At most 1 Gaussian source x1 and x2 Gaussian, uncorrelated, and unit variance Density function is completely symmetric Does not contain info on direction of the columns of the mixing matrix A.