Model Risk: Evaluation, Control, and Importance in Financial Institutions

1. Model Review & Model Risk [ChApter 8]

2. Sources • Federal Reserve ?comptroller of the Currency 2011 Supervisory Guidance on Model Risk Management • Massimo Morini, Understanding and Managing Model Risk (2011) • Emanuel Derman, “Markets and Models” (2001) • Riccardo Rebonato, “Theory and Practice of Model Risk Management” (2003)

3. How important is model risk? • Models are just convenient mathematical shorthand • Felix Salmon: “Li’s Gaussian Copula model will go down in history as instrumental in causing the unfathomable losses that brought the world financial system to its knees.” • Modeler quoted by Morini: “models were not a problem . The problem was in the data and the parameters! The problem was in the application!”

4. Model risk evaluation and control • Scope of model review and control • Roles and responsibilities • Model verification • Model validation • Continuous review • Periodic review

5. Model risk evaluation and control • Scope of model review and control • Broad definition • Methodology • Purpose • Vendor products • Roles and responsibilities • Model verification • Model validation • Continuous review • Periodic review

6. Model risk evaluation and control • Scope of model review and control • Roles and responsibilities • Business unit • Independent reviewer as “second set of eyes” • Effective challenge • External resources • Documentation standards • Senior management • Model verification • Model validation • Continuous review • Periodic review

7. Model risk evaluation and control • Scope of model review and control • Roles and responsibilities • Model verification • Independent implementation • Degenerate cases and extreme inputs • Graphs to build intuition • Benchmarking against previous model • Deal representation • Approximations • Model validation • Continuous review • Periodic review

8. Model risk evaluation and control • Scope of model review and control • Roles and responsibilities • Model verification • Model validation • Continuous review • Daily P&L reconciliation • Back-testing • Analysis of overrides • Periodic review

9. Model risk evaluation and control • Scope of model review and control • Roles and responsibilities • Model verification • Model validation • Continuous review • Periodic review • Changes in population of transactions • Changes in market environment • Cahnges in academic literature or market practices • Changes in technology

10. Model risk evaluation and control • Scope of model review and control • Roles and responsibilities • Model verification • Model validation • Interpolation approach • Cost of hedging approach • Prevailing market model approach • Matching model validation to model purpose • Liquid Instruments • Illiquid instruments • Trading models • Continuous review • Periodic review

12. Model Risk • Liquid positions • Models may be used for interpolation between liquid price quotes • Models will be used for calculation of the impact of market changes on positions in the VaR and stress test calculations • These model uses are robust and easy to test, since they can constantly be checked against actual liquid market quotes • Only need to be concerned with current Greeks (you can always ask that positions be reduced in future in reaction to changing Greeks) • Illiquid positions • Model becomes critical and hard to validate • Must employ as much liquid market data in the model as possible to avoid unnecessary staleness of MTM • Must clearly identify illiquid inputs and estimate liquidation risk through conservative assumptions (relative to net exposure to illiquid inputs) • Model Greeks are only useful in representing sensitivities to liquid market data; 1212they cannot be used in identifying the potential cost of being wrong about the illiquid inputs (since your risk on the illiquid inputs is not to small hedgable changes) • Even for Greeks related to liquid market data, must be concerned about future evolution, since you may not be able to change positions easily

13. The Li Model • Is it fair to blame the Li model for the collapse of the subprime mortgage CDO market? • The major firms whose losses led to the largest part of the crisis were all using more sophisticated models than the Li model • The Li model was used in very much the same way for CDOs as the BSM model is successfully used for vanilla options, with market-implied correlation skews playing a parallel role to market-implied volatility surfaces • The big difference is liquidity: the CDO market does not possess somewhat liquid instruments with characteristics that approximate the CDOs needing to be hedged • The Li model could be used to reasonably accurately compute the dependence of tranches on undiversifiable risk and to produce stress test results

14. Options position risk • Because of the wide variety of options contract specifications, there needs to be some underlying principle for integrating risk reporting • Puts and calls • Variety of dates that options expire • Variety of strikes at which options can be exercised • The Black-Scholes model is the key to managing and reporting options position risk (see Chapter 11, particularly Section 11.1 and 11.7) • It is used to interpolate market prices for all options from reported prices for a handful of options • It is used to identify a few key variables that impact the prices of all options • It is used to measure the change in options prices to changes in these key variables • There are known flaws in the model but risk managers and traders have developed procedures for dealing with them

15. Flaws in the Black-Scholes formula • Trivial flaws • BS assumes that assets prices are log-normally distributed when in fact they have fat tails • Any distribution can be accommodated by a slightly more complex model and closely approximated by BS using different volatilities at different strike levels • BS assumes a constant risk-free interest rate when in fact rates vary by maturity date • Variability of rates can be accommodated by basing BS on the price of a forward contract on the asset • BS assumes that hedging will take place continuously • Experience and simulations show that reasonably frequent hedging is very nearly as effective as continuous hedging

16. Flaws in the Black-Scholes formula • Significant flaws concerning hedging • BS assumes that hedging can take place without transaction costs • Large options trading desks only need to hedge net exposures which holds transaction costs down to a very small part of overall trading costs • Other users of options can count on large options trading desks to eliminate pricing discrepancies • BS assumes that asset prices follow a smooth path with no sudden jumps • Risk reports need to be developed to show exposure to price jumps • By engaging in both buying and selling options, this exposure can be kept under control

17. Flaws in the Black-Scholes formula • Significant flaws concerning volatility • BS assumes that volatility is known in advance • Risk reports need to be developed to show exposure to volatility uncertainty • By engaging in both buying and selling options, this exposure can be kept under control • BS assumes that volatility is constant when in fact volatility varies by time period and price level • Risk reports on volatility exposure need to show details of exposure bucketed by tenor and strike

18. Option position reporting using the Black-Scholes Greeks • The “Greeks” are the sensitivities of an option portfolio based on changes to the inputs to the BS formula • Sensitivities that need to be consolidated with non-option positions • Delta is the sensitivity to a change in asset price • Delta is used to consolidate option exposures into position reports for the asset • Very important in making sure that undiversifiable exposure to stock indices and government bond price levels is captured • Rho is the sensitivity to changes in interest rates • Rho is used to consolidate option exposures into interest rate position reports • Two interest rate positions may be needed (e.g, Dollars and Euros on an FX option) • Sensitivities that are unique to options • Vega is the sensitivity to changes in volatility • Exposure to a parallel shift in volatility is the most important risk exposure that is unique to options • Theta is sensitivity to changes in time to expiry • Theta reports show exposure to time decay – getting closer to expiration of the option with no further price moves • Gamma is the sensitivity of delta to a change in asset prices • Gamma reports are a first approximation to exposure to jumps in asset prices

19. Option position reporting that goes beyond the Greeks • Price-volatility matrix reporting • Shows the exposure of an options book that is currently delta-hedge (hedged against small changes in asset prices) to different combinations of large changes in asset price and volatility • Very important for getting a complete picture of exposure to price jumps (look at Table 11.6 to see a portfolio with 0 gamma and 0 vega that still has exposure to price jumps) • Vega bucket reporting • The principal exposure to volatility changes is picked up by the vega measure • Basis risks between options of different maturities requires bucketing vega by time to expiry • Basis risks between options of different strikes requires bucketing vega by strike

20. Price-volatility matrix for a call

21. Price-volatility matrix for a call-spread (“risk reversal”)

22. Price-volatility matrix for a calendar spread

23. How do we know that these fixes to BSM can be used to control risk? • Historical experience shows there have been no major blow-ups due to market-making in reasonably liquid vanilla options • Simulations can be performed to show that while risk has not been eliminated it has been reduced to controllable levels (see section 11.3 for details) • Adequate risk control requires frequent (but not continuous) trading in a liquid underlying asset and infrequent trading in a less liquid (but somewhat liquid) set of options with strike and tenor characteristics that reasonably approximate the options being hedged

24. Dynamic hedging strategies

25. Dynamic hedging simulation

26. The Li Model • Is it fair to blame the Li model for the collapse of the subprime mortgage CDO market? • The major firms whose losses led to the largest part of the crisis were all using more sophisticated models than the Li model • The Li model was used in very much the same way for CDOs as the BSM model is successfully used for vanilla options, with market-implied correlation skews playing a parallel role to market-implied volatility surfaces • The big difference is liquidity: the CDO market does not possess somewhat liquid instruments with characteristics that approximate the CDOs needing to be hedged • The Li model could be used to reasonably accurately compute the dependence of tranches on undiversifiable risk and to produce stress test results

27. Calculation of the Liquidation Margin for Product with Sporadic Liquidity • Need to build a model that relates pricing on this product to products with full liquidity • Must change MTM continuously based on changes to these related products, using the model • Estimate the degree of added liquidation risk caused by the uncertainty of the modeled relationship • The key rule is that the illiquid product must show an added degree of liquidation risk above that of the more liquid proxy product • The model is relatively easy to check, since it can be recalibrated every time a new sporadic quote becomes available

28. Calculation of the Liquidation Margin for Position of Illiquid Size • Illiquidity of position size must be judged relative to normal trading volume • Can estimate liquidation risk by VaR-like price moves over a time period in which orderly liquidation can take place • Alternatively, can model the market impact of large volume • In either case, this is an adjustment for added cost of liquidation beyond the normal liquidation costs of a smaller position • Need to be concerned with the impact of changing Greeks throughout the liquidation period

29. The nature of actuarial risk • Actuarial instrument risk can arise from • Instruments with one-way market – instruments for which almost all customer interest is on one side of the market and ability to lay off risk is limited by all dealers having similar positions (you can’t take comfort from price quotes that can’t be acted on) • Instruments that require extensive information disclosure and negotiation to realize anything other than fire-sale prices • Instruments that can only be liquidated under restrictive conditions

30. The nature of actuarial risk • Actuarial risk requires a different approach from management, traders and marketers than market risk does • Longer time horizon – willingness to wait until sale or expiry to fully recognize gains • Willingness to live with larger degree and duration of uncertainty of results • Capital structure that allows for patience • Different approach to assuring adequate capital • Actuarial risk is not for suitable for everyone!!!

31. Countering pro-cyclicality of regulatory capital requirements • Pro-cyclicality can be eliminated by the ability to sell liquid assets and by using a fixed target for non-liquid assets • There are two ways to deal with the pro-cyclical problem of higher demands for capital during economic downturns, when it is hardest to raise capital: • Positions can be liquidated • Adequate capital can be raised prior to the economic downturn • This implies that for liquid positions, capital can be tied to changes from current mark-to-market. Even though this will result in higher capital requirements during economic downturns, the ability to liquidate positions makes this manageable. • For illiquid positions, capital should not be tied to mark-to-market. In an economic downturn, capital will rise severely because losses on MTM cost capital on top of rising capital requirements tied to increased volatility. The option of position liquidation is not available to relieve this bind. [Sections 8.4.4 and 5.5.8.1]

32. Regulatory capital for illiquid positions • The alternative for illiquid positions is capital requirements tied to a fixed stress-level (e.g., default levels three times historical averages). This causes adequate capital to be raised when assets are first booked. When an economic downturn occurs, the capital drain of MTM losses is balanced by reduction of capital requirements, since current MTM is closer to the fixed stress-level. • Accounting standards should impact disclosure to shareholders and lenders, not regulatory capital standards • This is a natural outcome of the recommendation to tie required capital to fixed stress-level losses. • De-coupling regulatory capital requirements from earnings reporting would ease regulatory and investor concerns that building up reserves during good times is a way of manipulating the reporting of earnings volatility

33. Using liquid proxies to estimate illiquid instrument risk Use liquid proxies to represent these trades in computations designed for the firm’s liquid positions, such as MTM, VaR and stress tests Avoids stale MTMs Measures risk concentrations and encourages diversification Every model is a potential liquid proxy representation; every liquid proxy representation is a model subject to error Model difference between actual product and liquid proxy to create conservative valuation assumptions Modeling of differences must go all the way to final payout and must reflect the possibility that the model used for pricing and trading the product may be wrong Modeling should be by Monte Carlo simulation to reflect a full range of possible outcomes The liquid proxy is just a form of representation; it is not intended to dictate hedging action to the trading desk Traders who believe that they have better hedging strategies should be given sufficient limit room to act on these (plausible) beliefs and should reap the resulting gains (and losses)

34. Estimating illiquid instrument risk • Derman: “Because of their illiquidity, many of these positions [in long-term or exotic over-the-counter derivative securities that have been designed to satisfy the risk preferences of their customers] will be held for years. Despite their long-term nature, their daily values affect the short-term profit and loss of the banks that trade them.” • Rebonato: “What differentiates trading in opaque instruments from other trading activities is the possibility that the bank might accumulate a large volume of aggressively-marked opaque instruments. When, eventually, the true market prices are discovered, the book-value readjustment is sudden, and can be very large. Stop-loss limits are ineffective to deal with this situation, since the gates can only be shut once the horse has well and truly bolted.” • Derman: “Derivative models work best when they use as their constituents underlying securities that are one level simpler and one level more liquid than the derivative itself.” (my emphasis)

35. Derman on estimating illiquid instrument risk • “It’s never clear what profit and loss will result from hedging a derivative security to its expiration. Markets will move in unexpected ways, sometimes intensifying transactions costs and often dismantling what seemed a reasonable hedging strategy. These effects are rarely captured by the conventional models used in front-office valuation systems.” • “Therefore, for illiquid positions, it is important to estimate the adjustments to conventional marked values that can occur as a result of long-term hedging. One should build Monte Carlo models that simulate both underlyer behavior and a trader’s hedging strategy to create distributions of the resultant profit or loss of the whole portfolio. These distributions can be used to determine a realistic adjustment to the trading desk’s conventional marks that can be withheld until the trade is unwound and their realized profit or loss determined…Monte Carlo analysis provides a good sense of the variation in portfolio value that will be exhibited over the life of the trade due to transactions costs, hedging error and model risk. Ultimately, such analyses should be part of the desk’s own front-office valuation system.”

36. Rebonato on estimating illiquid instrument risk • “Model risk is the risk of occurrence of a significant difference between the mark-to-model value of a complex and/ or illiquid instrument, and the price at which the same instrument is revealed to have traded in the market.” • “From the perspective of the risk manager the first and foremost task in model risk management is the identification of the model (‘right’ or ‘wrong’ as it may be) currently used by the market to arrive at traded prices.” • “…market intelligence and contacts with the trader community at other institutions are invaluable” • Requires a variety of models to reverse-engineer observed prices • Requires information about as many observed prices as possible • “The next most important task of the risk manager is to surmise how today’s accepted pricing methodology might change in the future.” (including changes to model, changes to calibration, and changes to numerical implementation). • “Being aware of the latest market developments and of academic papers can be very useful in guessing which direction the market might evolve tomorrow.” • “To a large extent, the model risk management task can be described as an interpolation and extrapolation exercise that simply cannot be carried out in an informational vacuum [without anchor points of solid knowledge about the levels and nature of actual market transactions]”

37. Integrating Derman’s approach & Rebonato’s approach Derman emphasizes simulation of long-term hedging while Rebonato emphasizes anticipating changes in pricing methodology due to model changes and calibration changes. Are these contrasting approaches or two aspects of the same general methodology? • Simulation of long-term hedging is a way to calculate the financial impact of changes in model calibration. Can it also handle changes in model specification? • Pro: Since you are simulating all the way through to final payout, model changes are irrelevant • Con: “…the sudden occurrence of large-notional trades for which it is difficult to establish a clear rationale on the basis of customer-driven demand” or a shift in market structure (e.g., exit of a large dealer) could potentially force liquidation of a portfolio prior to final payout, at which point the prevailing model will be very relevant. • Can you calculate the impact of potential changes in pricing methodology without simulation of long-term hedging? • Maybe you could just reprice a portfolio with a range of potential models and calibrations. • But since you are vulnerable to the impact of model and calibration changes not just at the present time but over time, including changes in price levels, it’s hard to see how this can be done without some form of long-term simulation.

38. Estimating Illiquid Instrument Risk • Model difference between actual product and liquid proxy to create conservative valuation assumptions • Modeling of differences must go all the way to final payout and must reflect the possibility that the model used for pricing and trading the product may be wrong • Modeling should be by Monte Carlo simulation to reflect a full range of possible outcomes • Derman recommends a full simulation that includes both underlying behavior and trader hedging strategy. This represents an ideal that may sometimes be difficult to achieve in practice. • Simulation by assuming infrequent rehedging can make use of more public information and can be easier to implement, but at a cost of greater conservatism, since the full range of trader hedging strategies will not be captured.

39. Liquid proxy example: variance swaps [Section 12.1.1] Some non-liquid instruments can be so closely estimated by liquid proxies that the residual risk is quite small These instruments are sometimes called “quasi-vanillas” The Breeden-Litzenberger theorem says that any option whose payout is a smooth function of a terminal price can be perfectly replicated by an (infinite) package of forwards and plain vanilla calls and puts In practice, a finite package can be used to replicate closely, with degree of residual risk easily estimated Variance swaps (swaps with payout based on the square of realized volatility) are popular instruments, since they can be used to express a vie on volatility that is independent of price level Variance swaps can be shown to be precisely replicated by delta hedging in forwards plus an option with payout based on the logarithm of terminal price

40. Liquid proxy example: Illiquid swap [Section 10.2.2] • Suppose there is a liquid two-way market in a certain currency’s swaps out to 7 years, but only a one-way market to pay 10 year fixed • A full simulation of a hedging strategy could be quite complex. • An alternative is to use a 7-year swap as the liquid proxy for the 10-year swap and conservatively estimate the illiquid piece as the cost of a one-time trade, 3 years from now, to reverse a 4-year swap and put on a 7-year swap (this is called a “stack-and-roll”) • Every day for which public liquid quotes of 4 and 7 year swap rates are available contributes an independent data point for estimating this cost • If the liquid market only extends to 5 years, you would need two trades: a 2 year swap into a 5 year swap in 3 years, followed by a 3 year swap into a 5 year swap after 2 more years • This approach generalizes to illiquid maturity options, though a range of strikes needs to be used to offset the impact of price changes prior to the one-time trade date and the full vol-surface (skew as well as time) needs to be considered

41. Liquid proxy example: digital options [Section 12.1.4] Delta hedging of a digital option can be very dangerous, given the reality of discrete hedging. When the market is near the strike close to expiry, the delta hedge called for could result in very large losses Full simulation of an optimal delta hedging strategy would be very difficult, depending on hard-to-determine information about the liquidity of large trades An alternative is to use a call spread (selling a call at a strike lower than the digital’s strike and buying a call at a strike above the digital’s strike) as a proxy for selling a digital option Estimate the illiquid piece as the difference between the potential payout on the digital and payout of the call spread Hypothetical size of the call spread depends on how wide the strikes are set – the narrower the difference in strikes, the larger is the required size The representation by the liquid proxy allows correct representation in risk reports showing sensitivity to volatility level and volatility skew. The closer the strikes for the call spread, the smaller the potential loss on the illiquid piece. Selection of the strikes for the call spread needs to be wide enough to make for liquid trade size and manageable deltas and gammas as the trade evolves Risk of the illiquid piece can be easily estimated by Monte Carlo simulation. If you have a large book of digital options, negative impacts for some digitals on a given Monte Carlo path may be offset by positive impacts for some other digitals.

42. Liquid proxy example: barrier options [Section 12.3.3] Suppose there is a one-way market for selling a down-and-out call option (a call option that has zero payoff if the asset price goes below a certain barrier any time during the life of the option) A liquid proxy would be selling a European call option with the same strike and expiry date as the barrier option along with the purchase of European put options struck below the barrier If the barrier is never struck, the puts expire worthless Estimate the illiquid piece by the cost of reversing the European call and put positions at the time the barrier is hit, relative to a set of scenarios Each scenario would specify the time the barrier is hit, the level of implied volatility at the time the barrier is hit, the volatility skew at the time the barrier is hit, and the slope of the forward price curve (the drift) at the time the barrier is hit The representation by the liquid proxy allows correct representation in risk reports showing sensitivity to volatility level and volatility skew. This is similar to Peter Carr’s approach (and gives the same results when volatility skew and drift are zero) but may utilize more expiry dates for the puts and explicitly simulates hedge slippage costs for varying volatility skew and drift

43. Liquid proxy example: barrier options (continued) Every day for which public liquid quotes for the volatility surface of the European options is available contributes an independent data point for estimating the cost of the illiquid piece An approach in line with Derman’s recommendation is to pick the best model you can for evolution of prices, incorporating stochastic volatility and the possibility of jumps, and use Greeks to identify the liquid proxy. But it then requires much more work (Monte Carlo simulation of recalibration of the Greeks) to calculate the potential cost of difference between the barrier trade and the liquid proxy You can’t just utilize current Greeks of non-liquid inputs, such as jump frequency, since these might not be stable Very important: must be consistent in liquid proxy choice and simulation of cost of liquidity. If you are going to change the liquid proxy based on model calibration, then you must use the more detailed simulation method! This approach can easily be extended to other exotic options such as double barriers, partial barriers, lookback options, and compound options

44. Liquid proxy example: illiquid CDO tranche[Section 13.4] Create a set of scenarios corresponding to different states of the economy, i.e., correlation is enforced through a principal component. Use a model that fully incorporates exact modeling of cash flows in a given scenario. Do a complete simulation to price each CDO tranche in each of these scenarios, assuming a fully diversified portfolio (i.e., no residual correlation beyond the principal component). Probability weights can be assigned to scenarios based on observed prices of the underlying basket and observed prices of liquid CDO tranches (if available). The model can be used to determine a good hedging portfolio, utilizing the underlying basket and liquid CDO tranches . This hedging portfolio can be used as a liquid proxy in risk reports. Model risk can be measured by a Monte Carlo simulation of final payouts of the illiquid CDO tranche plus the hedging portfolio, including the impact of lack of full diversification. Model risk must be judged by final payouts to maturity without any assumption of ability to sell a CDO. A full simulation would include periodic recalculation of the hedging portfolio. A more conservative, but more easily computable, simulation would assume the initial hedge is held throughout.

45. Derman & Wilmott on What Makes a Good Model • Physics, because of its astonishing success at predicting the future behavior of material objects from their present state, has inspired most financial modeling…Financial theory has tried hard to emulate physics and discover its own elegant, universal laws. But finance and economics are concerned with the human world of monetary value. • There are no fundamental laws in finance. And even if there were, there is no way to run repeatable experiments to verify them. Financial theories written in mathematical notation—aka models—imply a false sense of precision. Good modelers know that. • What makes a good model, then? We think the best ones use only a few variables and are explicit about their assumptions. In this regard, we believe that the Black-Scholes model for options valuation—now often maligned—is a model for models. It is clear and robust. Clear because it is based on true engineering: It gives you a method for manufacturing an option out of stocks and bonds, and it tells you what, under ideal circumstances, the option should be worth…The world of markets doesn't exactly fulfill the ideal conditions Black-Scholes requires. But the model allows an intelligent trader to see what real-world dirt has been swept under the rug—and to adjust his or her risk estimates accordingly. • I will never sacrifice reality for elegance without explaining why I have done so. Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.

46. Derman on What Makes a Good Model Review • One should…ask the following set of open ended questions: • Does the model embody an accurate description of…the derivative’s payoffs? • Does the model provide a realistic (or at least plausible) description of the factors that affect a derivative’s value?...All models are simplifications…Is the assigned model an appropriate simplification? • Has the model been appropriately calibrated to the observed behavior, parameters and prices of the simpler, liquid constituents that comprise the derivative? • Is the software reliable? • Model documentation: Since software and models often have long lives…documentation provide[s] a long-term corporate memory of the principles and implementation of the model. • Comprehensive price verification: Nothing is better than a completely independent check of price and hedge ratios. A strategist knowledgeable about the market, but organizationally separate from the trading desk, should start with the confirmed trade details and build an independent model… • Periodic comprehensive model review: …as markets mature and market participants gain experience of the supply, demand and shocks that their underlyers and derivatives can experience, prices often change character and start to reflect these realities more accurately….It is therefore advisable to periodically revisit entire derivative markets and their models…