Pricing Model Risk & Control

1. Pricing Model Risk & Control Ragveer Brar Valuation & Controls Team September 2013 The views expressed in this presentation are personal and do not necessarily reflect those of BoE or PRA.

2. Contents • The Role of Models in Derivative Pricing • Front to Back Valuation of Derivatives • Model Risk Management Principles • Model Development • Model Implementation • Model Usage • Model Inventory • Valuation Uncertainty • Model Risk and IPV • Background Documents / References

3. The Role of Models in Derivative Pricing What Are Models for • Banks rely heavily on models in most aspects of financial decision making. They routinely use models for a broad range of activities, including underwriting credits; valuing exposures, instruments, and positions; measuring risk; managing and safeguarding client assets; determining capital and reserve adequacy; and many other activities. • Our presentation today will focus on models used in derivative pricing. • Black-Scholes Formula changed the game, by making it faster and more efficient to price risk. They also tied pricing and risk management to a specific, simplistic mathematical framework, which does not well represent real-world dynamics for most instruments. • More and more complex models developed over time: - Hull-White - SABR - Gaussian Copula - and many more

4. The Role of Models in Derivative Pricing Model Inputs and Outputs • Derivative pricing models – Mathematical formulae that transform a number of input parameters into prices for financial derivatives, often based on some assumptions. • Inputs – Parameters: visible vs. invisible; e.g. volatility of underlying, mean reversion, correlation etc. Assumptions: explicit vs. implicit; e.g. risk free discount rate is LIBOR, efficient market hypothesis • Outputs – Prices: pricing and trading; fair value in financial statements Risks: risk management; hedging • “All models are wrong but some are useful.” The use of derivative pricing models inevitably lead to valuation uncertainty. Parameters Assumptions underlying Models volatility Prices strike risk free rate Quantification of Risks

5. Front to Back Valuation of Derivatives What Are Models for • The valuation process for derivatives is complicated, with various roles and responsibilities shared by different areas of a bank. Each step in the process brings its own risks, and requires specific controls to ensure accurate balance sheet values. • The steps can be broken down into 3 main areas; Model Development, Model Implementation and Model Usage. For each of these areas specific elements need to be well executed, and strongly controlled by an independent function:

6. Model Risk Management Principles FED / OCC Guidance • The Supervisory Guidance on Model Risk Management issued in April 2011 by Board of Governors of the Federal Reserve System (SR 11-7) and the Office of Comptroller of the Currency (OCC 2011-12) provides updated regulatory guidance governing the use of models at all US national banks. • The graph summarises the framework outlined in the Guidance. • The principles contained therein are broadly applicable to banks outside the US and should be considered by other regulators and market participants. * Graph taken from The evolution of model risk management, May 2013

7. Model Risk Management Principles FED/OCC Guidance – Key Messages • Firms should adopt a holistic model risk management framework, spanning all three lines of defence. • Model validation should be seen as a risk management function, and not as a compliance function. • Model risk should be primarily mitigated at source through increased formality in Front Office control frameworks for model development, implementation and usage. • Residual model risk should be mitigated by independent control functions, through more real-time model validation and on-going model monitoring. • Developing and maintaining strong governance, policies and controls over the model risk framework is fundamentally important to its effectiveness. • A bank’s internal audit function should assess the overall effectiveness of the model risk management framework.

8. Model Development Control Environment OCC/FED Guidance • “Model design, theory and logic should be well documented and supported by sound practice, with special attention to a models weaknesses and limitations”. • “Comparison with alternative theories and approaches is fundamental to a sound modelling process”. • Rigorous data analysis should be performed to demonstrate that models are suitable for any given product-underlying mix. • Model assumptions should be explicitly stated, tracked and analysed so that users are aware of potential limitations. • Model testing should be performed to assess a model’s accuracy, stability and behaviour in a range of scenarios, including those outside of normal expectations, such as negative interest rates, or extremely high volatility. Extreme values should be used to test limiting assumptions. • Model validation should be performed by an independent function, with the requisite knowledge, skills and expertise to assess the models, as well as the explicit authority to challenge developers. Additional Comments • The validation process should test every element of the model development process.

9. Model Development Common Deficiencies • Model weaknesses and limitation are often largely ignored when documenting new models. • Alternative economic theories are very rarely actively explored. Those theories that are explored are generally just minor tweaks on the same theme. Alternative models are often not used to compare model results. • The quality of data analysis is patchy and inconsistent, e.g. very few model validation teams consider a model’s expected P&L performance. • Model assumptions are often buried in economic theory or mathematical derivation, and are not stated clearly, considered by validation, or tracked post-development. • Model testing is often limited to ‘reasonable’ scenarios, and does not test a model’s ability to hand stressed market conditions. • Reporting lines for model validation teams are often unclear due to join/multiple reporting lines. The reporting line of model validation is inconsistent across industry. • Model validation often do not test significant parts of the model development process.

10. Model Implementation Control Environment OCC/FED Guidance • Sound systems should be maintained “to ensure data and reporting integrity, together with controls and testing to ensure proper implementation of models, effective systems integration and appropriate use”. • Implementation of vendor or third party systems should also be subject to formal model validation. Additional Comments • Specific validation should be performed for each implementation of each element of a model, with special connection to the interactivity between the elements, e.g. implementation of a new engine should require testing of the impact on all pricers which use that engine. • Production code should be checked to ensure that the model that has been implemented matches the model that has been proposed and validated. • Model libraries should be controlled by independent control functions, and any changes to models should require formal change protocols. Model developers should not have write access to production systems.

11. Model Implementation Common Deficiencies • Often no consideration is given to data integrity, or the ability to generate accurate reports when implementing new models, resulting in problems analysing and controlling model inventory, or errors in model outputs. • Validation is often restricted to specific elements of a model, with other elements taken for granted, e.g. where a pricer has worked with one engine or system, it is assumed that it will work with another. • Surprisingly often, lack of sufficient checks results in models being implemented which are erroneous, or different from that proposed by model developers. • Model libraries are often controlled by the Front Office, with changes not being subject to formal controls. In some cases, developers have access to production systems as well as development systems. • Third party systems are often taken for granted, with very little formal validation performed by the bank implementing the system.

12. Model Usage Control Environment • For each product, there should be one officially approved model. Model Users (Front Office) should be explicitly restricted in their ability to select alternative models. • “Scripts” (highly bespoke, low volume models) should be individually validated, and subject to tight restrictions and limitations. • “Model parameters” (model specific pricing inputs, typically updated infrequently) such as mean reversion speeds or barrier shifts should be subject to monitoring, and independent validation in the same way as other pricing inputs. • Calibration sets, such as the structure of yield curves and volatility surface should be locked down, with changes to the sets following model change protocols. • Calibration results should be independently validated (IPV) as part of the valuation control process. • Firms should take a broad view of what counts as a model and requires formal model validation. This should include “model adjustments” required to match market pricing, and reserve calculations (such as bid-offer reserves).

13. Model Usage Common Deficiencies • Flexibility is often given to Front Office in model choice, which can result in cherry picking of models in order to maximize valuation / minimize capital. • “Scripts” are often over-used by traders, don’t exist in core pricing systems, and are not well validated or controlled. • Model Parameters are sometimes excluded from formal controls, unmonitored by independent functions and not subject to regular validation. • We have seen cases of calibration sets being completely under the control of front office, with the impact of changes not monitored or controlled independently. This has been used by Front Office as a tool to mismark their portfolios. • Model adjustments are often completely controlled by Front Office, and outside of the formal oversight of independent controllers. These adjustments have built up to be highly material (>£1b) without meaningful challenge by controllers. • Reserve methodologies are often set by Front Office, without any validation by independent control, despite their material impact on balance sheet values of derivatives. Alternatively, they are set by Product Control, but without any of the model development or validation standards applied to other valuation models used in the firm. • Various weaknesses exist in IPV processes across all firms, but they are too many to cover in this session.

14. Model Inventory Control Environment OCC/FED Guidance • Market dynamics change over time, sometimes dramatically (as seen in interest rate markets in 2008 / 2009). All models should be subject to on-going monitoring, and periodic revalidation to ensure they maintain acceptable levels of performance, accuracy and stability. Additional Comments • Firms should have a model risk framework that attempts to model risk and ensures that senior management are aware of the materiality of the uncertainty this creates. • Restrictions and limitations raised during the model validation process should be centrally monitored, and used to set threshold conditions for formal model reviews, and to ensure Front Office compliance with model control policies. • Excessive model adjustments suggest that a model does not accurately represent the market dynamic for a particular product set. Model adjustments should form part of the formal monitoring and reporting on model risk. At a minimum model adjustments need to be calculated by an independent control function each month end, and ideally there should be strict thresholds at which adjustments trigger automatic model reviews, cuts in risk limits or trading curbs.

15. Model Inventory Common Deficiencies • Many firms lack any framework for modelling model risk. Where such frameworks exist, reporting is often poor and little management attention is given to the risks inherent in models. • Periodic review of models is often of a very low standard, with a large proportion of models going many years without any formal review. • Some firms lack any framework for monitoring model restrictions and limitations. Amongst those firms that do monitor restrictions, breaches are often ignored such that there are no repercussions for trading breaching restrictions. • Model adjustments are often allowed to build up in an uncontrolled manner, without any challenge from controllers, or consequences for front office activity booked on those models.

16. Valuation Uncertainty Valuation Uncertainty due to Pricing Models • Valuation Uncertainty is the existence of a range of plausible values for a position or portfolio of positions, at the reporting date and time. The recognition of the inherent valuation uncertainty fundamentally changes the derivative valuation from a point estimate to a range estimate. • Different pricing models and calibration methods can be used for valuing the same derivative contract; and different netting methodologies can be used for the same portfolio of derivatives, leading to different valuations and valuation adjustments. Valuation uncertainty assessment should include uncertainty due to pricing models. • In practice, building alternative models and setting up the full range of possible models for the full range of possible calibration methods and netting methodologies may be too intensive an approach to be adopted across all model risk quantification. • Where quantification of model risk by model comparison/re-calibration is not possible an institution should establish a framework for quantifying model risk with an associated uncertainty charging structure.

17. Valuation Uncertainty Regulatory Focus (1) • In the UK, GenPru 1.3 requires firms to use prudent valuation principles when valuing trading books (and other assets and liabilities held at fair value) and report their Prudent Valuation Adjustment (PVA) to the FSA/PRA through the Prudent Valuation Return (Policy Statement PS12/07). • One of the considerations for assessing Prudent Valuation Adjustment should be the assessment of model risk, which include but not limited to – Impact of using different models Impact of using different calibration methods Impact of using different maturity/strike buckets Impact of using different interpolation/extrapolation methods

18. Valuation Uncertainty Regulatory Focus (2) • In Europe, EBA Consultation Paper relating to Draft Regulatory Technical Standards on prudent valuation under Article 100 of the draft Capital Requirements Regulation was published in Jul 2013. • In the CP, model risk is a specific category of Additional Valuation Adjustment that need to be assessed by firms in the core approach. Institutions shall estimate a model risk AVA for each valuation model by considering valuation model risk which arises due to the potential existence of a range of different models or model calibrations. • In addition, the CP requires at least an annual review of valuation model performance. There is also a requirement to validate the use of netting schemes for reducing calculations for derivative exposures. This requirement arises from the concerns on the lack of model calibration for reserve requirements and risk netting. • Globally, regulators from other countries are also starting to focus on valuation risk.

19. Model Risk and IPV Control Environment • IPV should not just focus on input testing. Output testing should be performed, where possible, for complex derivatives and even for vanilla derivatives. Output testing should be used to calibrate model reserves and quantify model limitations. • Where output testing is not possible, input testing completeness should be performed thoroughly to ensure no “hidden” parameter remains untested. • IPV reporting should incorporate valuation uncertainty. Model reserves and their uncertainty should also be reported. Where products are only input tested, this should be explicitly shown in management packs to heighten awareness as to the scope of potential model risk.

20. Model Risk and IPV Common Deficiencies Independent Price Verification (IPV) is performed by firms to independently validate the valuations by Front Office. IPV and model control are intricately linked, through model inputs and outputs and model reserves as fair value adjustments; • Input vs. outputs IPV: - Many banks test input parameters ONLY, e.g. forward points, volatilities, dividend yields etc. - An effective completeness check is often lacking. All parameters tested? All assumptions tested? - There can be “hidden” input parameters not tested, e.g. mean reversion, volatility “shift” etc. - There can be implicit assumptions not tested, e.g. some CMS swap pricing models assume perfect replication by vanilla swaptions and therefore no other risks, such as a volatility shift factor used by traders. - Output testing often not performed. Where output testing is not feasible, there is often no reporting and no consideration for the implication on valuation uncertainty. • Incorporating model uncertainty in IPV reporting: - Lack of framework for effective management reporting that links the IPV and valuation uncertainty, including model uncertainty. - Lack of framework for linking model reserves to output IPV testing in management reporting.

21. Background Documents / References • Volatility and Correlation 2nd Edition, Rebonato • Supervisory Guidance on Model Risk Management, Office of the Comptroller of the Currency (http://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf) • Guidance on Model Risk Management, Federal Reserve Board (http://www.federalreserve.gov/bankinforeg/srletters/sr1107.pdf) • Global Model Practice Survey 2011, Deloitte (http://www.deloitte.com/view/en_GB/uk/industries/financial-services/f2640dfa4d9a5310VgnVCM1000001a56f00aRCRD.htm) • EBA Consultation Paper - Draft Regulatory Technical Standards (http://www.eba.europa.eu/-/eba-consults-on-draft-technical-standards-on-prudent-valuation) • FSA Policy Statement 12/7 - Regulatory Prudent Valuation Return, FSA (http://www.fsa.gov.uk/static/pubs/policy/ps12-07.pdf) • Product Control Findings and Prudent Valuation Presentation, FSA (http://www.fsa.gov.uk/pubs/other/pcfindings.pdf) • Model Risk Management Survey 2012, Pwc • The evolution of model risk management, Pwc (http://www.pwc.com/en_US/us/financial-services/regulatory-services/publications/assets/pwc-closer-look-model-risk-management.pdf)