1 / 39

Review of Probability

Review of Probability. Probability Theory:. Many techniques in speech processing require the manipulation of probabilities and statistics. The two principal application areas we will encounter are: Statistical pattern recognition. Modeling of linear systems. Events:.

acornelius
Download Presentation

Review of Probability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Review of Probability

  2. Probability Theory: • Many techniques in speech processing require the manipulation of probabilities and statistics. • The two principal application areas we will encounter are: • Statistical pattern recognition. • Modeling of linear systems.

  3. Events: • It is customary to refer to the probability of an event. • An event is a certain set of possible outcomes of an experiment or trial. • Outcomes are assumed to be mutually exclusive and, taken together, to cover all possibilities.

  4. Axioms of Probability: • To any event A we can assign a number, P(A), which satisfies the following axioms: • P(A)≥0. • P(S)=1. • If A and B are mutually exclusive, then P(A+B)=P(A)+P(B). • The number P(A) is called the probability of A.

  5. Axioms of Probability (some consequence): • Some immediate consequence: • If is the complement of A, then • P(0) ,the probability of the impossible event, is 0. • P(A) ≤ 1. • If two event A and B are not mutually exclusive, we can show that • P(A+B)=P(A)+P(B)-P(AB).

  6. Conditional Probability: • The conditional probability of an event A, given that event B has occurred, is defined as: • We can infer P(B|A) by means of Bayes’ theorem:

  7. Independence: • Events A and B may have nothing to do with each other and they are said to be independent. • Two events are independent if P(AB)=P(A)P(B). • From the definition of conditional probability:

  8. Independence: • Three events A,B and C are independent only if:

  9. Random Variables: • A random variable is a number chosen at random as the outcome of an experiment. • Random variable may be real or complex and may be discrete or continuous. • In S.P. ,the random variable encounter are most often real and discrete. • We can characterize a random variable by its probability distribution orby itsprobability density function (pdf).

  10. Random Variables (distribution function): • The distribution function for a random variable y is the probability that y does not exceed some value u, • and

  11. Random Variables (probability density function): • The probability density function is the derivative of the distribution: • and,

  12. Random Variables (expected value): • We can also characterize a random variable by its statistics. • The expected value of g(x) is written E{g(x)} or <g(x)> and defined as • Continuous random variable: • Discrete random variable:

  13. Random Variables (moments): • The statistics of greatest interest are the moment of X. • The kth moment of X is the expected value of . • For a discrete random variable:

  14. Random Variables (mean & variance): • The first moment, ,is the mean of x. • Continuous: • Discrete: • The second central moment, also known as the variance of p(x), is given by

  15. Random Variables …: • To estimate the statistics of a random variable, we repeat the experiment which generates the variable a large number of times. • If the experiment is run N times, then each value x will occur Np(x) times, thus

  16. Random Variables (Uniform density): • A random variable has a uniform density on the interval (a, b) if :

  17. Random Variables(Gaussian density): • The Gaussian, or normal density function is given by:

  18. Random Variables (…Gaussian density): • The distribution function of a normal variable is: • If we define error function as • Thus,

  19. Two Random Variables: • If two random variables x and y are to be considered together, they can be described in terms of their joint probability density f(x, y) or, for discrete variables, p(x, y). • Two random variable are independent if

  20. Two Random Variables(…Continue): • Given a function g(x, y), its expected value is defined as: • Continuous: • Discrete: • And joint moment for two discrete random variable is:

  21. Two Random Variables(…Continue): • Moments are estimated in practice by averaging repeated measurements: • A measure of the dependence of two random variables is their correlation and the correlation of two variables is their joint second moment:

  22. Two Random Variables(…Continue): • The joint second central moment of x , y is their covariance: • If x and y are independent then their covariance is zero. • The correlation coefficient of x and y is their covariance normalized to their standard deviations:

  23. Two Random Variables(…Gaussian Random Variable): • Two random variables x and y are jointly Gaussian if their density function is : • Where

  24. Two Random Variables(…Sum of Random Variables): • The expected value of the sum of two random variables is : • This is true whether x and y are independent or not • And also we have :

  25. Two Random Variables(…Sum of Random Variable): • The variance of the sum of the two independent random variable is : • If two random variable are independent, the probability density of their sum is the convolution of the densities of the individual variables : • Continuous: • Discrete:

  26. Central Limit Theorem • Central Limit Theorem (informal paraphrase): If many independent random variable are summed, the probability density function (pdf) of the sum tends toward the Gaussian density, no matter what their individual densities are.

  27. Multivariate Normal Density • The normal density function can be generalized to any number of random variables. • Let X be the random vector, • Where • The matrix R is the covariance matrix of X (R is Positive-Definite)

  28. Random Functions : • A random function is one arising as the outcome of an experiment. • Random function need not necessarily be functions of time, but in all case of interest to us they will be. • A discrete stochastic process is characterized by many probability density of the form,

  29. Random Functions : • If the individual values of the random signal are independent, then • If these individual probability densities are all the same, then we have a sequence of independent, identically distributed samples (i.i.d.).

  30. mean & autocorrelation • The mean is the expected value of x(t) : • The autocorrelation function is the expected value of the product :

  31. ensemble & time average • Mean and autocorrelation can be determined in two ways: • The experiment can be repeated many times and the average taken over all these functions. Such an average is called ensemble average. • Take any one of these function as being representative of the ensemble and find the average from a number of samples of this one function. This is called a time average.

  32. ergodicity & stationarity • If the time average and ensemble average of a random function are the same, it is said to be ergodic. • A random function is said to be stationary if its statistics do not change as a function of time. • Any ergodic function is also stationary.

  33. ergodicity & stationarity • For a stationary signal we have: • Where • And the autocorrelation function is :

  34. ergodicity & stationarity • When x(t) is ergodic, its mean and autocorrelation are :

  35. cross-correlation • The cross-correlation of two ergodic random functions is : • The subscript xy indicates a cross-correlation.

  36. Random Functions (power & cross spectral density): • The Fourier transform of (the autocorrelation function of an ergodic random function) is called the power spectral density of x(t) : • The cross-spectral density of two ergodic random functions is :

  37. Random Functions (…power density): • For an ergodic signal x(t), can be written as: • Then from elementary Fourier transform properties,

  38. Random Functions (White Noise): • If all values of a random signal are uncorrelated, • Then this random function is called white noise • The power spectrum of white noise is constant, • White noise is a mixture of all frequencies.

  39. Random Signal in Linear Systems : • Let T[ ] represent the linear operation; then • Given a system with impulse response h(n), • A stationary signal applied to a linear system yields a stationary output,

More Related