Loss Reserving: Performance Testing and the Control Cycle

1. Loss Reserving:Performance Testingand the Control Cycle Casualty Actuarial Society Pierre Laurin

2. Today’s agenda • Defining the problem • Performance testing — in general and in the context of reserves • The reserving actuarial control cycle • Case study — real-world results • This presentation is based on the paper “Loss Reserving: Performance Testing and the Control Cycle”, authored by Yi Jing, Joseph Lebens, and Stephen Lowe, that has recently been submitted for publication in Variance

3. THE PROBLEM Questions for the reserving actuary • How do you know that the methods you are currently using are the “best”? • What evidence supports your selection of methods? • What are the right weights for combining the results of the methods? • How do you decide when to change methods? • What is the confidence range around estimates? • How do you evaluate the cost/benefit of developing new data sources or implementing more complex methods? • How do you instill hubris in junior actuaries?

4. THE PROBLEM The Quiz Raw Scores of Respondents • Objective: To test respondents understanding of the limits of their knowledge • Respondents were asked to answer ten questions related to their general knowledge of the global property/casualty industry • For each answer, respondents were asked to provide a range that offered a 90% confidence interval that they would answer correctly • Ideally (i.e., if “well calibrated”), respondents should have gotten nine out of ten questions correct Number of Respondents The results of our research illustrate the prevalence of actuarial overconfidence Tillinghast Confidence Quiz Note: based on 374 respondents as of 4/5/04.Profile of respondents: 86% work in P/C industry; 73% are actuaries.

5. THE PROBLEM Reserves are forecasts! • An actuarial method is used to develop a forecast of future claim payments • An actuarial method consists of • An algorithm • A data set • A set of intervention points • The actuary must • Choose a finite set of methods from the universe M • Choose a set of weights to combine the results of each method together • Performance testing, via a formal control cycle, can help the actuary make these choices

6. PERFORMANCE TESTING Performance testing is a formal analysis of prediction errors • Test a particular method by running the method on historical data – comparing estimates from the method with actual run-off • Performance testing is a formalized process, not just a numerical exercise

7. PERFORMANCE TESTING The general framework for performance testing is cross-validation • Cross-validation in the context of a regression model

8. PERFORMANCE TESTING Criteria for assessing the performance of a reserving method: BLURS-ICE

9. THE CONTROL CYCLE Performance testing of reserving methods can be part of an institutionalized control cycle The Actuarial Control Cycle for the Reserving Process

10. CASE STUDY Case Study: U.S. Insurer • Commercial Auto BI data • 1972 to 1998 accident years – June 30th valuations • Paid and incurred counts and amounts • Estimates of claim liabilities from 1979 to 1998 – twenty years • December 31st valuation used as “actual ultimate” • Environmental influences during the period add difficulty to estimation • Economic and social inflation • Operational changes in claim department • Changes in underwriting posture

11. CASE STUDY Skill can be measured by comparing actual to predicted unpaid loss ratios • We could just use a paid B-F with a constant ELR and payment pattern every year. This would give us the dotted line as our estimate. • We are looking for methods that do better than a crude, constant assumption

12. CASE STUDY Formal measurement of skill • The skill of a method is measured by: • Where • mse = mean squared error • msa = mean squared anomaly • Skill is the proportion of variance “explained” by the method

13. CASE STUDY Actuarial methods subjected to performance testing

14. CASE STUDY Skill can be measured by maturity • Note that skill can be negative (e.g., paid loss projection method at 6 months), implying that it induces volatility rather than explaining it

15. For a given correlation, the optimal weights are those with the minimum combined variance Minimum starts at the very right, when correlation is 100% Minimum gradually shifts leftward as correlation decreases CASE STUDY The minimum-variance weighting of methods depends on their variances and their correlation

16. CASE STUDY Key conclusions from case study • Accuracy, or skill, can be formally measured, using cross-validation approach • Performance testing can guide the actuary in choosing methods, and selecting weights between methods; correlation is a relevant criteria in selecting methods and weights • Chain-ladder methods are seriously degraded by changing claim policies and procedures; using claim counts to adjust improves skill • Experimentation with non-traditional methods is worthwhile; case reserve development has higher skill at later maturities

17. CONCLUSION Good reasons to do performance testing • Opportunity to improve accuracy of estimates • Formal rationale for selected actuarial methods • Empirical validation of stochastic reserve risk models • Input to development of reserve ranges • Manage actuarial overconfidence • Cost / benefit of enhancements to data and systems