Predictive Modeling Spring 2005 CAMAR meeting

1. Predictive ModelingSpring 2005 CAMAR meeting Louise Francis, FCAS, MAAA Francis Analytics and Actuarial Data Mining, Inc www.data-mines.com

2. Objectives • Introduce Predictive modeling • Why use it? • Describe some methods in depth • Trees • Neural networks • Clustering • Apply to fraud data

3. Predictive Modeling Family

4. Why Predictive Modeling? • Better use of insurance data • Advanced methods for dealing with messy data now available

5. Supervised learning Most common situation A dependent variable Frequency Loss ratio Fraud/no fraud Some methods Regression CART Some neural networks Unsupervised learning No dependent variable Group like records together A group of claims with similar characteristics might be more likely to be fraudulent Some methods Association rules K-means clustering Kohonen neural networks Major Kinds of Modeling

6. Two Big Specialties in Predicative Modeling

7. Modeling Process Internal Data Data Cleaning External Data Deploy Model Other Preprocessing Build Model Validate Model Test Model

8. Data Complexities Affecting Insurance Data • Nonlinear functions • Interactions • Missing Data • Correlations • Non normal data

9. Kinds of Applications • Classification • Prediction

10. The Fraud Study Data • 1993 Automobile Insurers Bureau closed Personal Injury Protection claims • Dependent Variables • Suspicion Score • Number from 0 to 10 • Expert assessment of liklihood of fraud or abuse • 5 categories • Used to create a binary indicator • Predictor Variables • Red flag indicators • Claim file variables

11. Introduction of Two Methods • Trees • Sometimes known as CART (Classification and Regression Trees) • Neural Networks • Will introduce backpropagation neural network

12. Decision Trees • Recursively partitions the data • Often sequentially bifurcates the data – but can split into more groups • Applies goodness of fit to select best partition at each step • Selects the partition which results in largest improvement to goodness of fit statistic

13. Goodness of Fit Statistics • Chi Square CHAID (Fish, Gallagher, Monroe- Discussion Paper Program, 1990) • Deviance CART

14. Goodness of Fit Statistics • Gini Measure  CART • i is impurity measure

15. Goodness of Fit Statistics • Entropy C4.5

16. An Illustration from Fraud data: GINI Measure

17. First Split All Claims p(fraud) = 0.36 Legal Rep = Yes P(fraud) = 0 .612 Legal Rep = No P(fraud) = 0.113

18. Example cont:

19. Example of Nonlinear FunctionSuspicion Score vs. 1st Provider Bill

20. An Approach to Nonlinear Functions: Fit A Tree

21. Fitted Curve From Tree

22. Neural Networks • Developed by artificial intelligence experts – but now used by statisticians also • Based on how neurons function in brain

23. Neural Networks • Fit by minimizing squared deviation between fitted and actual values • Can be viewed as a non-parametric, non-linear regression • Often thought of as a “black box” • Due to complexity of fitted model it is difficult to understand relationship between dependent and predictor variables

24. The Backpropagation Neural Network

25. Neural Network • Fits a nonlinear function at each node of each layer

26. The Logistic Function

27. Universal Function Approximator • The backpropagation neural network with one hidden layer is a universal function approximator • Theoretically, with a sufficient number of nodes in the hidden layer, any continuous nonlinear function can be approximated

28. Nonlinear Function Fit by Neural Network

29. Interactions • Functional relationship between a predictor variable and a dependent variable depends on the value of another variable(s)

30. Interactions • Neural Networks • The hidden nodes pay a key role in modeling the interactions • CART partitions the data • Partitions capture the interactions

31. Simple Tree of Injury and Provider Bill

33. Missing Data • Occurs frequently in insurance data • There are some sophisticated methods for addressing this (i.e., EM algorithm) • CART finds surrogates for variables with missing values • Neural Networks have no explicit procedure for missing values

34. More Complex Example • Dependent variable: Expert’s assessment of liklihood claim is legitimate • A classification application • Predictor variables: Combination of • claim file variables (age of claimant, legal representation) • red flag variables (injury is strain/sprain only, claimant has history of previous claim) • Used an enhancement on CART known as boosting

35. Red Flag Predictor Variables

36. Claim File Variables

37. Neural Network Measure of Variable Importance • Look at weights to hidden layer • Compute sensitivities: • a measure of how much the predicted value’s error increases when the variables are excluded from the model one at a time

38. Variable Importance

39. Testing: Hold Out Part of Sample • Fit model on 1/2 to 2/3 of data • Test fit of model on remaining data • Need a large sample

40. Testing: Cross-Validation • Hold out 1/n (say 1/10) of data • Fit model to remaining data • Test on portion of sample held out • Do this n (say 10) times and average the results • Used for moderate sample sizes • Jacknifing similar to cross-validation

41. Results of Classification on Test Data

42. Unsupervised Learning • Common Method: Clustering • No dependent variable – records are grouped into classes with similar values on the variable • Start with a measure of similarity or dissimilarity • Maximize dissimilarity between members of different clusters

43. Dissimilarity (Distance) Measure • Euclidian Distance • Manhattan Distance

44. Binary Variables

45. Binary Variables • Sample Matching • Rogers and Tanimoto

46. Results for 2 Clusters

47. Beginners Library • Berry, Michael J. A., and Linoff, Gordon, Data Mining Techniques, John Wiley and Sons, 1997 • Kaufman, Leonard and Rousseeuw, Peter, Finding Groups in Data, John Wiley and Sons, 1990 • Smith, Murry, Neural Networks for Statistical Modeling, International Thompson Computer Press, 1996

48. Data Mining CAMAR Spring Meeting Louise Francis, FCAS, MAAA Louise_francis@msn.com www.data-mines.com