How Wrong is your Model? Efficient Quantification of Model Risk

1. How Wrong is your Model?Efficient Quantification of Model Risk Advanced Statistical Methods in Credit Risk Royal Statistical Society Alan Forrest, RBS Group London, 13th June 2013 Information Classification – PUBLIC

2. Disclaimer • Thanks • With many thanks to the Credit Research Centre, University of Edinburgh Business School for supporting me as a Visiting Scholar November and December 2012; and to the Royal Bank of Scotland Group for granting me Special Leave to visit the CRC. • Disclaimer • The opinions expressed in this document are solely the author’s and do not necessarily reflect those of The Royal Bank of Scotland Group or any of its subsidiaries. • Any graphs or tables shown are based on mock data and are for illustrative purposes only.

3. Overview • Model Risk Principles and Background • Model Risk – an emerging and powerful idea in bank regulation and credit risk model management. • Model risk assessment needs quick and quantified sensitivity analysis. • Geometry and Information • Model specification and model sensitivity analysis can be presented geometrically within a classical, mathematically rich theory. • Efficient Sensitivity Analysis • Insight to the modelling process. • Practical strategies for sensitivity analysis and managing model risk.

4. Model Risk Background • The US Regulator (Fed / OCC) • “The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.” • The Statisticians (George Box, Norman Draper) • “Essentially, all models are wrong, but some are useful.” • The Logicians (attr John Maynard Keynes, but in fact Carveth Read) • “It is better to be roughly right than precisely wrong.”

5. Model Risk Background • FSA - Turner Review - March 2009 • “Misplaced reliance on sophisticated maths” • Too many pricing and lending decisions did not take into account the assumptions and limitations of the models. • BoE - The Dog and the Frisbee – Haldane, August 2012 • “… opacity and complexity…It is close to impossible to tell whether results from [internal risk models] are prudent.” • If we cannot say why we trust a model, are we right to use it? • Fed / OCC 2011-12a • “Model Risk should be managed like other types of risk.”

6. Model Risk Background • Sensitivity Analysis • How different would the model be if … ? • The missing data were filled in in a different way • A different time-period was used to develop the model • We enter a downturn or stressed period • An expert panel decision happened to turn out differently • Etc. • This is the key to quantitative model risk, but it is hard work • Then combinations of these sensitivities… • Model fitting is a multi-stage process… • Sensitivity impact (eg capital, provisions) is complex and multidimensional… • Is this work out of proportion to the benefits? • Can we do sensitivities quickly? Without refitting models?

7. Geometry and Information • Describing Data • A model is a description of the development data. • Model developers chose one of these descriptions for use in a decision. • Model Risk considers the degree to which this choice could influence the decision. • This talk considers frequency histogram or contingency table descriptions of models and data (Kullback, Centsov). • Discrete buckets with sample frequencies (proportions, not counts). • This is not a practical limitation: data elements are often classified and always finite precision. • We should not restrict too early the breadth of models that could be chosen. • This defines a high but finite-dimensional dimensional space whose coordinate values are the frequency in each bucket. • The observed data is a single point in this space. • Models are also “data” points: points that are preferred for use or descriptive convenience. • Dimension is the number of buckets, with one constraint: sum to 1 .

8. Geometry and Information • Illustration of a 3-cell model space: 2 dimensional simplex

9. Model chosen “closest to the data” Data Model Subspace Geometry and Information • Data, model subspace and model choice

10. Original data Modified data Geometry and Information • Sensitivity, model space flatness, over-sensitivity and over-fitting. Just right Models “Type 3 (or type 0) Error” Right answer to the wrong question Over-fitting Over-sensitive / discontinuous

11. Geometry and Information • Fitting a model to data: a log-linear example mw = ceaw Prior MLE fit Data

12. Geometry and Information • Likelihood maximisation and information • Some notation • d is the data point – vector (dw) of frequencies, indexed by bucket labels w , Sw dw =1. • m is the model point (mw) • N is the sample size • Log Likelihood of d, given m, is log L = - N *( I(d,m) + H(d) ) whereI(d,m) = Sw dw log(dw / mw) Kullback-Leibler divergence H(d) = - Sw dw log(dw), Shannon Entropy of the data. • Maximising likelihood is the same as minimising KL divergence. • but KL divergence is not a metric. • Following this line gives principles for practical inference, model selection etc.. • Akaike’s Information Criterion follows from this point of view. The Bayes Information Criterion also follows, assuming different priors and sample asymptotics.

13. Geometry and Information • Dual optimisation principles • For exponential families of models: • The blue spaces are linear. • The model space is generated “tropically”. • Principle of Maximum Likelihood • The model that maximises likelihood is where the data’s blue space hits the red model space. • Principle of Maximum Entropy (uniform prior) • Within each blue space, the red point maximises entropy, H(x) . • Principles of Inference • if m is the MLE fit to data d, and m’ is another model from the model space, then I(d,m’) = I(d,m) + I(m,m’) .

14. Geometry and Information • Two ways of moving distributions – taxic and tropic – shifts and tilts • Tropic movements generated by groups characterise exponential families of models Tropic movement f(x)  g(x) f(x) A group of functions acting multiplicatively on probability distributions. Renormalising after each action Taxic movement f(x)  f(x-t) Uses a group action on the state space.

15. Geometry and Information • Dual Foliations (Amari, Critchley et al.) • Assume a tropic structure. • Equivalently, decide what direction to give the blue subspaces. • Each model space is generated tropically from a prior m0 . • Maximum likelihood principle still applies. • Modified principle of maximum entropy: • Within each blue space, red points minimise I(x,m0) . • Principle of Inference still holds • Euclidean metric ds2 = Sw dxw2 is not the natural geometry.

16. Geometry and Information • Spherical Geometry • Another metric ds2 = Sw dxw2 / xw appears more natural. • Locally equivalent to KL divergence: ds2 = 2 I(x,x+dx) = 2 I(x+dx,x) , up to third order terms. • “Local Chi-squared”. • “Boot-strapping geometry”. • The model fitting foliations are orthogonal in this metric. • The model space curvature is true. • Isometric to a portion of a sphere. • 2x = u2 connects this space isometrically with Euclidean space.

17. Data shifted Model shifted Efficient Sensitivity Analysis • Data shifts give rise to model shifts • Seek a distance constraint: ds2(model) <= Const * ds2(data) • Seek understanding of the map: d(data)  d(model) • d(SQRT(data))  d(SQRT(model)) • d(LOG(data))  d(LOG(model)) • Linear, singular • Expansive or contractive directions? • Twists? Data Model

18. Efficient Sensitivity Analysis • More generally, seek conditions for contractive sensitivities. • These are the “flat” directions in the model space – good for model risk. • Some model families contract for all data-shifts, eg a model that groups two classes. • In context of logistic models, situations where sensitivities contract include: • Full interaction descriptions, eg population shifts (i.e. independent of default event); • Class amalgamation in dummy factors.

19. Efficient Sensitivity Analysis • Some general results are possible: • For an exponential family of models generated by dummy factors from a uniform prior: If m is the MLE model fitted to data d, then ds2(model) <= eI(d,m) ds2(data) . • Outline of proof: • A sequence of MLE models take us from d to m, whose nested model spaces drop one dimension at each step, d=m0 … mt=m. • The conditions assumed allow us to arrange that each step in the model sequence removes one dummy factor at a time. • The simplified 2-by-K context allows the direct calculation ds2(mi+1) <= eI(mi,mi+1) ds2(mi) • However, inference and the MLE conditions give us I(d,m) = I(d,m1) + I(m1,m2) + … + I(mk-1,m) • QED • Conjecture this is true for general exponential families and general priors.

20. Scale = Chi-sq (df = dimension) / (2N) Data Bootstrapping Ellipsoid = ds-ball Model Prediction Error Ellipsoid = ds-ball Model Standard Error Ellipsoid = image of data ds-ball Efficient Sensitivity Analysis • A sensitivity principle implied by boot-strapping • For large development samples, the standard error ellipsoid is sufficient to describe model sensitivity to data shifts.

21. Implications for Model Risk • Managing Sensitivities • Sensitivity analysis asks “what if the data was different?” Now we can connect model changes to data movements geometrically. • Sensitivities can be prioritised by the (most generous) metric distance of its specified data change, ds2(data) . • Sometimes it is simpler to prioritise on the divergence, which is numerically a good approximation to the distance. • Establish a limit to the size of data sensitivity, below which material changes of the model clearly will not happen. • Convert model variation, measured by ds2(model), into a simple business measure, as a communicable measure of model specification risk. • For example, conservatism equivalent, swap-set size, effect on gini, provisions impact or RWA impact.

22. Implications for Model Risk • Example • A model risk with default identification has been identified for a logistic scorecard model. A sensitivity analysis has been proposed: How different could the model be if the default counts were undercounted? • Assume a recorded 5% default rate, with possible additional defaults up to 5.5%. A general upper bound for data ds2 in this case is about 0.005 . • A sharper upper bound could be achieved knowing more about the factor distributions. • This prioritizes this sensitivity analysis in the model’s management. • All things being equal, it will come after a check on missing values accuracy (est. ds2 = 0.2) and before a check on the robustness of classing another factor (est. ds2 = 0.003). • With a sample size of 10,000 , and data space of dimension 100, this gives a 95%-ile bootstrap ds2 radius of about 0.01 (1/20000 *chi-sq(df=100) ). Therefore the sensitivity proposed is about 1/2 the strength of a standard error 95% significance movement. • Conclusion: assuming this model is built on robust use of standard errors, this sensitivity should cause no change to the significance of the factors selected, but could change the factor selection and the size of some of the parameters.

23. Conclusions • Model Risk Principles • Model risk is an important and growing area of banks’ risk management. • The key to quantitative model risk assessment is sensitivity analysis, and • The key to practical sensitivity analysis is a quick, effective method to gauge model variation without having to rebuild models. • Efficient Sensitivity Analysis and Model Risk Managment • Classical ideas of statistical geometry and information theory add insight to the process of model fitting, • Sensitivity analysis is framed as a differential data-shift problem. • This approach to sensitivity analysis develops fully general principles, and practical methods that cut resource and by-pass the need to refit models.