Model Risk Management Insights from Banking Book Credit Risk

1. Model Risk ManagementInsights from Banking Book Credit Risk 6th Annual Marcus Evans conference on Pricing Model Validation Alan Forrest, RBS Group Risk Analytics Independent Model Validation London, 9th September 2013 Information Classification – PUBLIC

2. Disclaimer • Disclaimer • The opinions expressed in this document are solely the author’s and do not necessarily reflect those of The Royal Bank of Scotland Group or any of its subsidiaries. • Any graphs or tables shown are based on mock data and are for illustrative purposes only.

3. Overview • Background • Banking Book Credit Risk background. • Model Risk – an emerging and powerful idea in bank regulation and credit risk model management. • Model Risk Management • Model risk is managed like any other risk. • Model risk assessment needs quick and quantified sensitivity analysis. • Sensitivity analysis can be managed and assessed with minimal need for detailed refitting of models.

4. Banking Book Credit Risk • The Banking Book • Loans and facilities to obligors and customers. • Intention to hold until maturity. • Default and Loss • Driven by the behaviour of each individual entity and the constraints of the contract, • Within a latent economy and market environment. • Banking Book credit loss uncertainties are dominated by • Imperfect knowledge of individual obligor circumstances; • Uncertainty about how obligors perceive and interact with their environment.

5. Banking Book Credit Risk Models • Operational Models • Relationship management – acquisition decisions, pricing, limit setting. • Recoveries management – write-off process, collections. • Regulatory Capital Models • PD, EAD and LGD are aggregated from obligor level to portfolio level, with correlations set by the regulator.

6. Model Risk Background • The US Regulator (Fed / OCC 2011-12a ) • “The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.” • FSA - Turner Review - March 2009 • “Misplaced reliance on sophisticated maths” • The assumptions and limitations of the models were not communicated adequately to the pricing and lending decision-makers. • BoE - The Dog and the Frisbee – Haldane, August 2012 • “… opacity and complexity…It is close to impossible to tell whether results from [internal risk models] are prudent.” • If we cannot say why we trust a model, are we right to use it?

7. Model Risk Background • An empirical model splits into two pieces • A deterministic algorithm – to describe the data roughly. • A probability distribution – to describe the difference between the algorithm’s output and the observed outcomes. • Specification Risk concerns the correctness of both these parts of the model. • Accuracy (Prediction Risk) concerns the narrowness of the distribution in part 2.

8. Model Risk Background • Fed / OCC 2011-12a • “Model Risk should be managed like other types of risk.” • This talk will focus on Specification risk: • the part of model risk connected with model selection; • Model risk also includes risks in model implementation, use and interpretation.

9. Example Model Risk • A hypothetical PD model development • A Basel 2 IRB compliant PD model is required for a growing secured lending portfolio, based on a logistic regression scorecard.

10. UK data Local Data UK Model Local Model Local Expert Panel Example Model Risk • A hypothetical PD model development (ctd) • The PD model is required for a non-UK region, but the local portfolio has only a small size and thin data. • Modellers address this difficulty by using a model built on UK data, and reweighting to the local data and context. • To add local knowledge to the process an Expert Panel of local portfolio managers is formed to advise on some parameter choices.

11. Example Model Risk

12. Example Model Risk

13. Example Model Risk • Many sensitivities can be explored by rebuilding or reconsidering the model on adjusted data.

14. Sensitivity Analysis • Sensitivity Analysis • How different would the model be if … ? • The missing data were filled in in a different way • A local factor was used instead. • An expert panel decision happened to turn out differently • Etc. • This is the key to quantitative specification risk, but is this work out of proportion to the benefits? • Can we do sensitivities quickly? Without refitting models?

15. d(data) Data shifted d(model) Model shifted Efficient Sensitivity Analysis • Model chosen “closest” to data • Can be made geometrically precise • Data shifts give rise to model shifts • Seek understanding of the map: d(data)  d(model) • Seek a distance constraint: ||d(model)|| <= Const * ||d(data)|| • The data shift problem is hard • Prioritise and scope the sensitivity analysis, roughly, by ||d(data)|| Data Model

16. Managing Sensitivities • Model Risk - example revisited • ||d(data)|| can be determined for data shifts caused by different missing value imputation techniques, different expert views etc. • A cut-off (derived from a bootstrapping principle) indicates whether the data shift is capable of causing model movements in excess of standard error. • In this example, this cutoff has been computed as 0.223 .

17. Conclusions • Model Risk Background • Model risk is an important and growing area of banks’ risk management. • The key to quantitative model risk assessment is sensitivity analysis, and • The key to practical sensitivity analysis is a quick, effective method to gauge model variation without having to rebuild models. • Efficient Sensitivity Analysis and Model Risk Management • Sensitivity analysis can be framed as a mathematically precise differential data-shift problem. • This approach to sensitivity analysis leads to practical methods that manage resource and by-pass the need to refit models.