1 / 45

Minimal sufficient statistic

Minimal sufficient statistic. A statistic defines a partition of the sample space of (X 1 , … , X n ) into classes satisfying T(x 1 , … , x n ) = t for different values of t. If such a partition puts the sample x = ( x 1 , … , x n ) and

lilian
Download Presentation

Minimal sufficient statistic

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Minimal sufficient statistic Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  2. A statistic defines a partition of the sample space of (X1, … , Xn ) into classes satisfying T(x1, … , xn ) = t for different values of t. If such a partition puts the sample x =(x1, … , xn) and y = (y1, … , yn) into the same class if and only if then T is minimal sufficient for  Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  3. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  4. Rao-Blackwell theorem Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  5. The Exponential family of distributions • A random variable X belongs to the (k-parameter) exponential family of probability distributions if the p.d.f. of X can be written • What about • N(,  2 ) ? • Po( ) ? • U(0, ) ? Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  6. For a random sample x = (x1, … , xn ) from a distribution belonging to the exponential family Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  7. Exponential family written on the canonical form: Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  8. Completeness • Let x1, … , xn be a random sample from a distribution with p.d.f. f (x; )and T = T (x1, … , xn ) a statistic • Then T is complete for if wheneverhT (T ) is a function of T such that E[hT (T )] = 0 for all values of  then Pr(hT (T )  0) = 1 • Important lemmas from this definition: • Lemma 2.6: If T is a complete sufficient statistic for  and h (T ) is a function of T such that E[h (T ) ] =  , then h is unique (there is at most one such function) • Lemma 2.7: If there exists a Minimum Variance Unbiased Estimator (MVUE) for  and h (T ) is an unbiased estimator for  , where T is a complete minimal sufficient statistic for  , then h (T ) is MVUE • Lemma 2.8: If a sample is from a distribution belonging to the exponential family, then (B1(xi ) , … , Bk(xi) ) is complete and minimal sufficient for 1 , … , k Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  9. Maximum-Likelihood estimation Consider as usual a random sample x = x1, … , xnfrom a distribution with p.d.f. f (x;  ) (and c.d.f. F(x;  ) ) The maximum likelihood point estimatorof  is the value of  that maximizes L( ; x) or equivalently maximizes l( ; x) Useful notation: With a k-dimensional parameter: Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  10. Complete sample case: If all sample values are explicitly known, then Censored data case: If some ( say nc)of the sample values are censored , e.g. xi< k1 or xi> k2 , then where Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  11. When the sample comes from a continuous distribution the censored data case can be written In the case the distribution is discrete the use of F is also possible: If k1 and k2 are values that can be attained by the random variables then we may write where Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  12. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  13. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  14. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  15. Too complicated to find an analytical solutions. Solve by a numerical routine! Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  16. Exponential family distributions: Use the canonical form (natural parameterization): Let Then the maximum likelihood estimators (MLEs) of 1, … , k are found by solving the system of equations Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  17. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  18. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  19. Computational aspects • When the MLEs can be found by evaluating • numerical routines for solving the generic equation g( ) = 0 can be used. • Newton-Raphson method • Fisher’s method of scoring (makes use of the fact that under regularity conditions: • ) • This is the multidimensional analogue of Lemma 2.1 ( see page 17) Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  20. When the MLEs cannot be found the above way other numerical routines must be used: • Simplex method • EM-algorithm • For description of the numerical routines see textbook. • Maximum Likelihood estimation comes into natural use not for handling the standard case, i.e. a complete random sample from a distribution within the exponential family , but for finding estimators in more non-standard and complex situations. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  21. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  22. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  23. Properties of MLEs Invariance: Consistency: Under some weak regularity conditionsall MLEs are consistent Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  24. Efficiency: Under the usual regularity conditions: (Asymptotically efficient and normally distributed) Sufficiency: Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  25. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  26. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  27. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  28. Invariance property  Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  29. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  30. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  31. i.e. the two MLEs are asymptotically uncorrelated (and by the normal distribution independent) Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  32. Modifications and extensions Ancillarity and conditional sufficiency: Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  33. Profile likelihood: This concept has its main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters. The same ML point estimator for  1 is obtained by maximizing the profile likelihood as by maximizing the full likelihood function Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  34. Marginal and conditional likelihood: Again, these concepts have their main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  35. Penalized likelihood: MLEs can be derived subjected to some criteria of smoothness. In particular this is applicable when the parameter is no longer a single value (one- or multidimensional), but a function such as an unknown density function or a regression curve. The penalized log-likelihood function is written Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  36. Method of moments estimation (MM ) Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  37. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  38. The method of moments point estimator of  = ( 1, … ,  k ) is obtained by solving for  1, … ,  k the systems of equations Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  39. Example Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  40. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  41. Method of Least Squares (LS) First principles: Assume a sample xwhere the random variable Xi can be written The least-squares estimator of  is the value of  that minimizes i.e. Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  42. A more general approach Assume the sample can be written (x, z ) where xirepresents the random variable of interest (endogenous variable) and zi represent either an auxiliary random variable (exogenous) or a given constant for sample point i The least squares estimator of  is then Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  43. Special cases The ordinary linear regression model: The heteroscedastic regression model: Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  44. The first-order auto-regressive model: Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

  45. The conditional least-squares estimator of  (given ) is Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

More Related