Model Governance and Model Risk Management: Risk Manager’s Perspective

1. Model Governance and Model Risk Management: Risk Manager’s Perspective Nikolai Kukharkin Quantitative Risk Control, UBS Measuring and Controlling Model Risk, New York, October 2011 DISCLAIMER The views and opinions expressed in this presentation are those of the author and may not reflect the views and opinions of UBS and should not be cited as being those of UBS.

2. What Can Go Wrong With Models? • More extensive policies, stricter regulations, and more comprehensive model risk management programs.

3. Kill All the Quants?... * • Risky Business on Wall Street: High-tech supernerds are playing dangerous games with your money TIME magazine, April 11, 1994 • Recipe for Disaster: The Formula That Killed Wall Street Wired Magazine, Feb 23, 2009 • The Minds Behind the Meltdown: How a swashbuckling breed of mathematicians and computer scientists nearly destroyed Wall Street WSJ, Jan 22, 2010 • Financial Crisis Can Be Traced to “the Quants” The Kansas City Star, Feb 22, 2010 *) Andrew W. Lo, “Kill All the Quants?: Models vs. Mania in the Current Financial Crisis”, 2009.

4. …Before They Are Born… Too large a proportion of recent "mathematical“ economics are mere concoctions, as imprecise as the initial assumptions they rest on, which allow the author to lose sight of the complexities and interdependencies of the real world in a maze of pretentious and unhelpful symbols. John Maynard Keynes, The General Theory of Employment, Interest and Money, 1935

5. …Or May Be Not? Myron Scholes, Risk Magazine, September 2011 • It should be a golden age for risk modeling and management • “One thing about a crisis is that it shakes old opinions and you start learning new things. I hope we do – I am very bullish on the future for quants.” • He warns against overreliance on models, and concedes they had a role in the crisis. But the common-sense reaction – embracing intuition, and rejecting the use of modeling and quantitative techniques – is also flawed, he argues… • Presumably you used your intuition in picking the model, and intuition can fail, too.

6. New FED/OCC Supervisory Guidance on Model Risk Management • Expands on existing regulatory guidance by broadening the scope beyond model validation to include all aspects of model risk management at all stages: model development, implementation, and use • Revises and expands model and model risk definition • Establishes comprehensive model risk management program requirements • More formalized and expanded model governance and controls • Increased standing of model risk management function: needs to be influential; have explicit authority to challenge model developers and users • Model validation • Introduces “effective challenge” standard • Key elements of comprehensive validation : • Evaluation of conceptual soundness • On-going monitoring • Outcomes analysis

7. Comprehensive Model Risk Management Program Requirements • The bar has been raised significantly with respect to the scope, formality, rigor, and prominence expected of banks’ model risk management programs. • “Model risk should be managed like other types of risk.” • Life-cycle view of model risk - model risk management framework is expected to include standards for model development, use, and maintenance to which all model owners, users, and other stakeholders will be held. • Broader roles and responsibilities – it is not just the responsibility of the model validation unit: model developers / owners, users, validators, senior management, internal audit • Model risk management is an on-going, continuous, process – not a periodic activity: • Monitoring model risks and limitations identified during development and validation • Monitoring and on-going validation of changes (i.e., products, exposures, activities, clients, or market conditions) that may impact model risks • Regular model performance monitoring (i.e., back-testing, benchmarking, sensitivity analysis, and stress testing) • Model risk reporting to senior management and the board of directors.

8. How Should Banks Respond? • Examiners expect a bank to perform a self-assessment against new regulatory guidance, and have a clear action plan for closing identified gaps. • Potential action plan items may include the following: • Revisions to policies and procedures • Revisions to roles and responsibilities • Organizational changes • Development of new standards and guidelines for model development, implementation, and use • Revised model inventories (including an inventory of model-specific risks and limitations) • Mappings of model risk mitigation controls against existing inventory of model risks and limitations • Creation / enhancement of on-going model monitoring processes • Creation / enhancement of model risk reporting • Additional model validation testing (e.g., vendor models) • Creation of annual model review process

9. Model Validation: What’s Next? • Financial industry obtains a significant share of revenue from products valued by mathematical models • Models are here to stay and reliance on them will only grow • Consequently, model risk is a topic of great, and growing, interest in the risk management arena • How to define it • How to measure it • How to manage it • Qualitatively reasonably well defined • Much less successfully quantified and even less successfully managed • What is expected from model validators and how is the role changing? • What are the key priorities for model validators?

10. Model Risk: Define and Manage A model can be defined as: A simplified description or representation of an entity or process, property, characteristic or behavior which cannot be represented or predicted with complete certainty.The output of a model is therefore an estimate or approximation. 1. Model risk is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. 2. Model risk is the risk of error due to a deficiency in design or implementation of a pricing model. In other words, model risk is the risk of occurrence of a significant difference between the mark-to-model value of a product and its fair value. Or more flexible definition by R. Rebonato 3. Model Risk is the risk of occurrence of a significant difference between the mark-to-model value of a complex and/or illiquid instrument, and the price at which the same instrument is revealed to have traded in the market. Note that neither “true” nor “fair” value is mentioned, i.e. ->>> Market is the king More sophisticated / realistic / correct model is not necessarily the best

11. Model Governance Process • Regulators on Model Risk: FSA “Model Risk contributes to overall valuation risk. Model validation and model risk management processes are important elements of any valuation control framework. Whilst effective model validation is fundamental, model validation, however good, does not remove model risk. Few firms have sufficiently well developed frameworks for articulating model risk tolerance, and measuring and controlling model risks within that tolerance. We believe a better defined and implemented model risk management framework could therefore feed into a better defined and implemented valuation risk-management framework.” - from FSA’s “Dear CEO” letter on Valuation and Product Control principles, August 2008 • Regulators on Model Risk: FED and OCC Supervisory Guidance on Model Risk Management, April 2011 Model risk should be managed like other types of risk. Banks should identify the sources of risk and assess the magnitude. Model risk increases with greater model complexity, higher uncertainty about inputs and assumptions, broader use, and larger potential impact. …With an understanding of the source and magnitude of model risk in place, the next step is to manage it properly.

12. Model Risk Management Framework Goal is to set up a framework to explicitly, fully, and dynamically account for model risk • Partially accounted for by: • Qualitative capture • Models certification • Periodic model risk review • Quantitative capture • Model reserves • Sensitivity analysis • Portfolio reviews

13. Model Risk Management Framework Governance: Independent Verification Unit (IVU) with the mandate for: - independent review and certification of valuation models - independent risk-based review of model-related risks, i.e., the risk that the model either through a deficiency in design or implementation produces faulty output. The aim of the certification process is to obtain the required level of comfort that the model in question is functioning in an accurate and appropriate way. Model performance monitoring process: - Full review of potential model risks and existing control processes across systems and product areas taking into account factors including materiality, model choice, model applicability - This is independent from running certification processes, it challenges existing model issues and includes actions that are or should be taken Frequency and depth of reviews: Regulators requested model verification updates of the “high-risk” to be done more frequently, annually, compared to previous 5-y cycle for all certifications.

14. Model Governance Process – High-Risk Models Definition of High-Risk Models Define dimensions and criteria to categorize models as “high risk” Definition is altered based on new information / past actions Capture of High-Risk Models Agree on review types Run / participate in the reviews throughout the year Review Filter all models according to preliminary criteria Working Groups of IVU product specialists adjust filtered product list and define necessary actions Actions based on Reviews Consolidate results of reviews already undertaken Take further actions if necessary

15. Definition - Model Risk Types The following major model risks types were defined 1. Model inconsistencies or approximations Inconsistencies in mathematical assumptions of the model or its implementation Model assumed fit for purpose, although does not fully capture some features The model may be used outside of its range of applicability 2. Model choice Model choice uncertainty - several models are available (and one is being used) Many solutions could satisfy the same constraints 3. Calibration, model parameters, and input data issues Multiple sets of parameters can satisfy market; multiple sets of calibration instruments available, sometimes model cannot fit all of them simultaneously; uncertainty in model input parameters 4. Controls (booking approximations + level of oversight) The control environment into which the model is being released. Level of oversight by other control groups and therefore the probability of an error being detected 5. Complexity Exotic features, number of inputs, and the importance of inter-relationships between them models assumptions/conditions 6. Model/Product maturity and level of standardization Maturity, liquidity and rate of change of the market

16. Capture – Model Risk Scores Model risk scoring process Start with the certified product list and link and rank it according to the risk scores in several dimensions as well as materiality of positions: • Each product is described in terms of “High=3”, “Medium=2” or “Low=1” Risk Scores for each of the following six risk factors (combined Risk Score is between 6 and 18): 1. Model inconsistencies or approximations 2. Model Choice 3. Calibration, model parameters, and input data issues 4. Controls (booking approximations + level of oversight) 5. Complexity (exotic features, number of inputs, models assumptions/conditions) 6. Model/product maturity and level of standardization • Capability to compare model risk between models / products across all areas • This list is used as a starting point to identify models/products which will be subject to the annual review • Additional win – not just a formal more frequent re-certification, but rather a review targeting specific features which make this product/model high-risk

17. Is Theoretical Value Fair? • Most of the issues listed in the previous section can also give rise to model fair value adjustments Fair value is the price that would be received to sell an asset or paid to transfer a liability in an orderly transaction between market participants at the measurement date (FAS 157) • Accounting standards demand that if the value of asset or liability is not directly observable but rather obtained from a model (“marked-to-model"), it needs to be further adjusted to bring it into closer alignment with the market fair value. Why is such adjustment needed? • Model can have a known bias/deficiency • There can be an uncertainty around the model generated value due to: • existence of the alternative models (i.e., no industry standard) • non-uniqueness of calibration • uncertainty around (unobservable) model inputs • Terms “Model Fair Value adjustments” and “Model reserves” are often used interchangeably, but they mean different things and should not be mixed up

18. What Do Accounting Standards Say About Model Reserves? They say… NOTHING Model reserves may serve as a proxy, intuitive way to account for perceived model risk Model FV adjustments aim at “fine-tuning” model generated number to bring it into a better alignment with the market price. FV adjustments make the “best educated guess” of where the market is Model risk attempts to assess the tails of the theoretical price distribution, expresses how far off our “best guess” might be from the realized price • Model risk arises from the uncertainty in model specification, be it the model parameters and/or inputs (i.e., function arguments), or the model (function) itself. When not observable, FV of an asset is a variable characterized by some probability distribution. While model FV adjustments attempt to pinpoint the center of such distribution, its higher moments are the domain of model risk. • NOTE: Frequent practice of using “parameter uncertainty” and sometimes “alternative model” reserves to create a “conservative cushion” roughly the size of perceived model risk, contradicts the accounting standards which concentrate on fair value.

19. Inherent Model Risk • Valuation uncertainty beyond model FV adjustments is the domain of (inherent) model risk • Can be viewed as “residual” model risk. With proper model validation in place, inherent model risk can be minimized but never eliminated • Needs to be measured, monitored, and managed • Deserves a place in overall risk management framework on a par with market and credit risk • Should be considered alongside the market and credit risks in allocating capital, making business decisions, and managing the trading positions

20. Final Observations • There is always a risk that a model can be “wrong” • In part model risk is a variety of operational risk, i.e. the possibility of a human error • However, there is also an inherent uncertainty due to the very nature of financial modeling • The purpose of model governance is to set up policies and procedures that: • 1) minimize operational risk - Achieved through model validation, periodic reviews, model change management, back-testing, etc. • 2) provide for measurement, monitoring, and management of inherent model risk (model uncertainty) - Requires recognition of model uncertainty’s role alongside market and credit risk, and devising and implementing methods, processes and systems for measuring capturing reporting and managing model risk • Therefore, model risk deserves a place alongside market and credit risk in making business decisions (e.g., in capital allocation) as well as in the risk management and reporting process.