Some standard univariate probability distributions

1. Some standard univariate probability distributions • Characteristic function, moment generating function, cumulant generating functions • Discrete distribution • Continuous distributions • Some distributions associated with normal • References

2. Characteristic function, moment generating function, cumulant generating functions Characteristic function is defined as expectation of the function - e(itx) Moment generating function is defined as (expectation of e(tx)): Moments can be calculated in the following way. Obtain derivative of M(t) and take value of it at t=0 Cumulant generting function is defined as logarithm of characteristic function

3. Discrete distributions: Binomial Let us assume that we carry experiment and result of the experiment can be “success” or “failure”. Probability of “success” is p. Then probability of failure will be q=1-p. We carry experiments n times. What is probability of k successes: Characteristic function Moment generating function: Find first and second moments

4. Discrete distributions: Poisson When number of trials (n) is large and probability of successes (p) is small and np is finite and tends to  then binomial distribution converges to Poisson distribution: Poisson distribution can be expected to describe the distribution an event that occurs rarely in a short period. It is used in counting statistics to describe of number of registered photons. Find characteristic and moment generating functions. Characteristic function is: What is the first moment?

5. Discrete distributions: Negative Binomial Consider experiment: Probability of “success” is p and probability of failure q=1-p. We carry out experiment until k-th success. We want to find probability of j failures. (It is called sequential sampling. Sampling is carried out until stopping rule is satisfied). If we have j failure then it means that we number of trials is k+j. Last trial was success. Then probability that we will have j failures is: It is called negative binomial because coefficients are from negative binomial series: p-k=(1-q)-k Characteristic function is: What is the moment generating function? What is the first moment?

6. Continuous distributions: uniform Simplest form of continuous distribution is the uniform with density: Distribution is: Moments and other properties are calculated easily.

7. Continuous distributions: exponential Density of exponential distribution has the form: This distribution has two origins. • Maximum entropy. If we know that random variable is non-negative and we know its first moment – 1/ then maximum entropy distribution has the exponential form. • From Poisson type random processes. If probability distribution of j(t) events occurring during time interval [0;t) is a Poisson with mean value t then probability of time elapsing till first event occurs has the exponential distribution. Let Tr denotes time elapsed untilr-th event Putting r=1 we get e(- t). Taking into account that P(T1>t) = 1-F1(t) and getting its derivative wrt t we arrive to exponential distribution This distribution together with Poisson is widely used in reliability studies, life testing etc.

8. Continuous distributions: Gamma Gamma distribution can be considered as generalisation of exponential distribution. It has the form: It is probability of time t elapsing befor r events happens Characteristic function of this distribution is:

9. Continuous distributions: Normal Perhaps the most popular and widely used continuous distribution is the normal distribution. Main reason for this is that that usually random variable is the sum of the many random variables. According to central limit theorem under some conditions (for example: random variables are independent. first and second and third moments exist and finite then distribution of sum of random variables converges to normal distribution) Density of the normal distribution has the form Another remarkable fact is that if we know mean value and variance only then random variable has the normal distribution. There many tables for normal distribution. Its characteristic function is:

10. Exponential family Exponential family of distributions has the form Many distributions are special case of this family. Natural exponential family of distributions is the subclass of this family: Where A() is natural parameter. If we use the fact that distribution should be normalised then characteristic function of the natural exponential family with natural parameter A() =  can be derived to be: Try to derive it. Hint: use normalisation fact. Find D() and then use expression of characteristic function and D() . This distribution is used for fitting generlised linear models.

11. Continuous distributions: 2 Normal variables are called standardized if their mean is 0 and variance is 1. Sum of n standardized normal random variables is 2 with n degrees of freedom. Density function is: If there are p linear restraints on the random variables then degree of freedom becomes n-p. Characteristic function for this distribution is: 2is used widely in statistics for such tests as goodness of fit of model to experiment.

12. Continuous distributions: t and F-distributions Two more distribution is closely related with normal distribution. We will give them when we will discuss sample and sampling distributions. One of them is Student’s t-distribution. It is used to test if mean value of the sample is significantly different from 0. Another and similar application is for tests of differences of means of two different samples are different. Fisher’s F-distribution is distribution ratio of the variances of two different samples. It is used to test if their variances are different. On of the important application is in ANOVA.

13. Reference Johnson, N.L. & Kotz, S. (1969, 1970, 1972) Distributions in Statistics, I: Discrete distributions; II, III: Continuous univariate distributions, IV: Continuous multivariate distributions. Houghton Mufflin, New York. Mardia, K.V. & Jupp, P.E. (2000) Directional Statistics, John Wiley & Sons. Jaynes, E (2003) The Probability theory: Logic of Science