STATISTICAL INFERENCE PART III

1. STATISTICAL INFERENCEPART III EXPONENTIAL FAMILY, FISHER INFORMATION, CRAMER-RAO LOWER BOUND (CRLB)

2. EXPONENTIAL CLASS OF PDFS • X is a continuous (discrete) rv with pdf f(x;), . If the pdf can be written in the following form then, the pdf is a member of exponential class of pdfs of the continuous (discrete) type. (Here, k is the number of parameters)

3. REGULAR CASE OF THE EXPONENTIAL CLASS OF PDFS • We have a regular case of the exponential class of pdfs of the continuous type if • Range of X does not depend on. • c() ≥ 0, w1(),…, wk() are real valued functions of  for . • h(x) ≥ 0, t1(x),…, tk(x) are real valued functions of x. If the range of X depends on , then it is called irregular exponential class or range-dependent exponential class.

4. EXAMPLES X~Bin(n,p), where n is known. Is this pdf a member of exponential class of pdfs? Why? Binomial family is a member of exponential family of distributions.

5. EXAMPLES X~Cauchy(1,). Is this pdf a member of exponential class of pdfs? Why? Cauchy is not a member of exponential family.

6. EXPONENTIAL CLASS and CSS • Random Sample from Regular Exponential Class is a css for . If Y is an UE of , Yis the UMVUE of .

7. EXAMPLES Let X1,X2,…~Bin(1,p), i.e., Ber(p). This family is a member of exponential family of distributions. is a CSS for p. is UE of p and a function of CSS. is UMVUE of p.

8. EXAMPLES X~N(,2) where both  and 2 is unknown. Find a css for  and 2 .

9. FISHER INFORMATION AND INFORMATION CRITERIA • X, f(x;), , xA (not depend on ). Definitions and notations:

10. FISHER INFORMATION AND INFORMATION CRITERIA The Fisher Information in a random variable X: The Fisher Information in the random sample: Let’s prove the equalities above.

11. FISHER INFORMATION AND INFORMATION CRITERIA

12. FISHER INFORMATION AND INFORMATION CRITERIA

13. FISHER INFORMATION AND INFORMATION CRITERIA The Fisher Information in a random variable X: The Fisher Information in the random sample: Proof of the last equality is available on Casella & Berger (1990), pg. 310-311.

14. CRAMER-RAO LOWER BOUND (CRLB) • Let X1,X2,…,Xnbe sample random variables. • Range of X does not depend on . • Y=U(X1,X2,…,Xn): a statistic; does’nt contain . • Let E(Y)=m(). • Let prove this!

15. CRAMER-RAO LOWER BOUND (CRLB) • -1Corr(Y,Z)1 • 0 Corr(Y,Z)21   • Take Z=′(x1,x2,…,xn;) • Then, E(Z)=0 and V(Z)=In() (from previous slides).

16. CRAMER-RAO LOWER BOUND (CRLB) • Cov(Y,Z)=E(YZ)-E(Y)E(Z)=E(YZ)

17. CRAMER-RAO LOWER BOUND (CRLB) • E(Y.Z)=mʹ(), Cov(Y,Z)=mʹ(), V(Z)=In() The Cramer-Rao Inequality (Information Inequality)

18. CRAMER-RAO LOWER BOUND (CRLB) • CRLB is the lower bound for the variance of an unbiased estimator of m(). • When V(Y)=CRLB, Y is the MVUE of m(). • For a r.s., remember that In()=n I(), so,

19. LIMITING DISTRIBUTION OF MLEs • : MLE of  • X1,X2,…,Xnis a random sample.

20. EFFICIENT ESTIMATOR • T is an efficient estimator (EE) of  if • T is UE of , and, • Var(T)=CRLB • Y is an efficient estimator (EE) of its expectation, m(), if its variance reaches the CRLB. • An EE of m() may not exist. • The EE of m(), if exists, is unique. • The EE of m() is the unique MVUE of m().

21. ASYMPTOTIC EFFICIENT ESTIMATOR • Y is an asymptotic EE of m() if

22. EXAMPLES A r.s. of size n from X~Poi(θ). • Find CRLB for any UE of θ. • Find UMVUE of θ. • Find an EE for θ. • Find CRLB for any UE of exp{-2θ}. Assume n=1, and show that is UMVUE of exp{-2θ}. Is this a reasonable estimator?

23. EXAMPLE A r.s. of size n from X~Exp(). Find UMVUE of , if exists.

24. Summary • We covered 3 methods for finding UMVUE: • Rao-Blackwell Theorem (Use a ss T, an UE U, and create a new statistic by E(U|T)) • Lehmann-Scheffe Theorem (Use a css T which is also UE) • Cramer-Rao Lower Bound (Find an UE with variance=CRLB)