Taking Uncertainty Into Account: Bias Issues Arising from Uncertainty in Risk Models

1. Taking Uncertainty Into Account:Bias Issues Arising from Uncertainty in Risk Models John A. Major, ASA Guy Carpenter & Company, Inc.

2. Example: Exponential Distribution • N=20 observations • T = sample mean; l=1 true mean • MLE EP curve: • q-exceedance point (PML, VaR) • X.01 = 4.605 actual

3. Sampling Distribution of T

4. Estimated PDFs

5. Client Questions • What is the 1 in 100-yr PML (1% VaR)? • What is probability of exceeding 4.605? • Can you give me an EP curve to answer these and similar questions? • Does sampling error affect the answer? • Can I get unbiased answers?

6. 3 Kinds of Bias • “dollar” or X-bias: • the average of PML dollar estimates • “probabilistic” or P-bias: • the average true exceedance probability of estimated PML points • “exceedance” or Q-bias: • the average estimated exceedance probability

7. Exponential MLE is X-unbiased

8. Exponential MLE is X-unbiased

9. Exponential MLE is P-biased • for small q • Expected actual risk is greater than nominal • Uncertainty increases risk!

10. Exponential MLE is P-biased

11. Correcting for P-bias • Predictive distribution • “Prediction interval” in regression • Mix randomness and uncertainty • integrate model pdf over parameter distribution • Exponential model: • Predictive result:

12. Predictive vs. Model Density

13. Which to use? • MLE curve is X-unbiased • no uncertainty adjustment, but... • on average, gets right $ answer • Predictive curve is P-unbiased • “takes uncertainty into account” and... • on average, reflects true exceedance pr • But they disagree... • and it gets worse...

14. Exponential MLE is Q-biased • for small q • Expected estimated risk is greater than the true risk (at the specified threshold) • Uncertainty now causes risk to be overstated!

15. Exponential MLE is Q-biased

16. Correcting for Q-bias • Minimum Variance Unbiased Estimator • standard procedure in classical statistics • Rao-Blackwell Theorem • Expectation of unbiased estimator, conditional on sufficient statistic • Exponential model: • MVUE result:

17. MVUE vs. Model Density

18. Paradox • Say we get an estimated T=1 (correct) • MLE says X.01=4.605, Pr{X>4.605}=1% • Predictive: X.01=5.179 is p-unbiased • risk is greater than MLE answer because impact of uncertainty • MVUE: Pr{X>4.605}=.69% is q-unbiased • risk is less because MLE tends to overstate exceedance probability

19. How the Paradox Arises

20. Conclusions • Uncertainty induces bias in estimators • Biases operate in different directions • depends on the question being asked • There is no monolithic “fix” for taking uncertainty into account • Predictive distribution fixes p-bias, • while making q-bias worse

21. Recommendations • First: Show modal estimates (MLE etc.) • Second: Show effect of uncertainty • Keep uncertainty distinct from randomness • Sensitivity testing w.r.t. parameters • Confidence intervals on estimators • Third: Adjust for bias only as necessary • Carefully attend to the question asked • Advise that bias adjustment is equivocal