Scenario Analysis to Validate Models

1. Scenario Analysis to Validate Models Martin Goldberg Senior Director Head of Model Quality Review Standard and Poor’s martin_goldberg@standardandpoors.com

2. The Usual Caveats This presentation expresses my own personal opinions and may not represent the views of Standard and Poor’s Rating Services. This is not intended to be a fully inclusive handbook. Risk management is not a simple task, and you will have to put thought and effort into it beyond attending a conference, if managing model risk is your goal. My experience is primarily with valuation, VaR, Op Risk, and Credit Risk models, so this may be less informative about validation of stat arb and portfolio construction models.

3. Outline of My Talk • Definitions of Terms • Using the appropriate scenarios for validating various types of financial models • Validation scenarios as distinct from model scenarios or Monte Carlo modeling • Benchmarking using alternative models • How do you correctly conduct scenario analysis? • What are its limitations? • Conclusions

4. Definition of Scenario • I define “scenario” as a (possibly hypothetical) time series of values and/or changes for all market factors, prices, rates, and other inputs to the relevant model, with defined start and end dates and defined intermediate time intervals. • The start date is usually “now” • Whether the scenario unfolds in microseconds, days, or years depends on the intended use of the model. • An input might have a time series of “unchanged” if you aren’t interested in that particular factor, which is equivalent to an explicit assumption that the effects of that factor on the model results are negligible or not of concern. • A “lesser” input could also be assumed to be some deterministic function of the “important” risk factors.

5. Definition of Scenario Analysis • I define “scenario analysis” as • choosing one or more scenarios, • running the relevant model in each scenario, and • studying (analyzing) the results of each scenario to draw conclusions. • The conclusion could be that you need more or different scenarios.

6. Definition of Model • I assume that this is limited to automated quantitative financial models, which I define as • A quantitative model is a controlled view of certain real world dynamics that is used to infer the likely consequences of some pre-specified assumptions under various circumstances. • May not capture all the nuances of the real world. • A quantitative calculation based on one or more assumptions. • Not black boxes of revealed truth but merely numerical expressions of some view of how the world would be likely to behave. • Relies on assumptions about the behavior of people, organizations, acts of the natural world, and the use of other models by market participants. • A mixture of behavioral psychology, statistics, numerical methods, and subjective opinions.

7. Definition of Model Validation • I define the validation of a financial model as a test of how suitable it is for its intended use, which involves a simultaneous test of assumptions, inputs, calibration, implementation, and usage. • The physical sciences have laws of nature called "theories," that observation or experiments can verify or disprove. • In finance, however, there are merely significant tendencies and patterns • Quantitative financial models are necessarily generalizations that events in the real world will sometimes contradict.. • Different assumptions and different intended uses will in general lead to different models. • Models intended for one use may not be suitable for other uses.

8. Appropriate scenarios – a menu • There are several ways of choosing validation scenarios, and often it is helpful to use more than one. • Historical scenarios – observed past intervals that might be a useful test for the model. This is also called backtesting. • Sensitivity tests – Here the start date and the end date are both now, and all but one (or perhaps a few) of the inputs are unchanged. (Instantaneous jumps) • Ladder tests – a set of sensitivity tests with various widely spaced values of the same input. • Monte Carlo – Generate lots of random scenarios, examine each individually, and see which ones cause the model to behave oddly or unexpectedly. Look for patterns. This is not the same as statistical analysis of the result distribution. • Hand-picked – What particular few scenarios are most instructive abut the model’s behavior and limitations? • Stress scenarios – any of the above where the changes are quite large

9. Appropriate scenarios for validating any model • The point of running scenarios for validation is to see how the model behaves, separately from any details about the positions that are being modeled. The size of portfolios being run through the model should, in my opinion, all be quite small, and at least as diverse as anything you plausibly might use the model for in the foreseeable future. • Scenarios are tests of the model’s response to market inputs, not to position inputs. In a long enough scenario, you may wish to include the feedback effects of portfolio or model changes that would be caused with certainty by embedded contract triggers or company policies.

10. Scenarios for validating Valuation Models • Try to include scenarios that just barely trigger, or just barely miss triggering, any knock-outs, contingencies, etc. in the instrument being modeled. • Look for scenarios that cause the model to “break” and try to determine where the breakpoints are. • Compare the model Greeks to the valuation outputs for small changes, at a variety of starting points for the scenarios. • For path-dependent valuations, the scenarios should explore a variety of different-shaped paths.

11. Scenarios for validating regression models • For regression-type models, such as are used in, for example, portfolio optimization, operational risk capital calculations, some forms of stat arb, and credit scorecards, scenario analysis other than historical backtesting may not be the best tool. • A regression on historical data would not give the same results if history had been different, and counterfactual hypothetical scenarios may not give the validation effort any useful information.

12. Scenarios for validating Portfolio Optimization and VaR model Inputs • The classic variance-covariance Markowitz framework used by some of these models can lend itself to scenario testing on the input pre-processing. • In this simulated example, the Gaussian drivers of two processes are 61% correlated. Consider scenarios where we test robustness to skewness in the distribution of one or both observed processes. A rank correlation remains stable, but the Pearson correlation is an underestimate of concordance. Skewness of equity indices: Australia is -2.8, US -1.2

13. Scenarios for validating predictive models • For some forms of predictive model, such as technical analysis, scenarios can be used to see how quickly the model perceives new trends or reversals, and how credible that perception is. • For a structured credit waterfall model, informative scenarios would extend to the structure’s final maturity, and try to “break” the model with various shapes of stressful paths of the relevant inputs.

14. Stress Test Scenario Example • Here is an example of a ladder test where the market moves outside the range of appropriateness • Calculate the forward implied volatility from given Black-Scholes implied vols from quoted options at 1 and 2 year tenors. Suppose the volatility of the 1 year option is 5%. The forward vol in a simple model (no jumps, no autocorrelation) would be • If the vol curve drops too quickly, as with some commodities, you need a more complicated model since imaginary forward vols are not meaningful Imaginary forward vols

15. Scenario Validation Stresses larger than model can handle appropriately Designed to test model Scenarios to test facets of model Examine each scenario result Choose a few representative positions Only report results that are informative or unexpected Calibrate if needed, and not necessarily to any particular date Scenarios usually ad-hoc, need not be plausible Scenarios might include regime changes Scenario-based model Stresses large but plausible OR Monte-Carlo with no plausibility test Designed for informative outputs Meaningful scenarios Possibly only use ensemble statistics Run entire portfolio through each scenario Report all results Periodic up-to-date recalibration Scenarios subject to senior management scrutiny Has market changed regimes enough to make model irrelevant or obsolete? Validation scenarios vs Scenario-based models

16. Benchmarking Using Alternative Models • The best validation technique, which takes the longest but gives the most assurance that a model does what it should, is to run the model in question, and a benchmark model, on many scenarios with various portfolios, and study where they get different results. • If the benchmark model is off-the-shelf software, the differences in results are likely due to different assumptions. • If the benchmark is a custom built replica of the model being validated, independently coded using the same specifications, then differences are either coding bugs or inconsistency with the specifications.

17. Benchmarking with Partial Alternatives • One way to test the model’s assumptions and features is to use a custom benchmark model where one or more features can be switched off or new ones added. • Run the model and this partial benchmark model, on many scenarios with various portfolios, and see how much each assumption leads to different results, and look for synergistic effects.

18. Correct conduct of scenario analysis – First few steps • This should be one of the later steps in the validation process, after you are comfortable that the code isn’t too buggy • Determine which inputs you will be varying in the scenarios, and if there are any “special” values – strikes, barriers, trigger points, etc. • Design some of the scenarios before you run any of them • This will help get buy-in from model users and builders if something unfortunate turns up in your testing • Ameliorate the governance issue of “Quants don’t run this bank” • More on this sort of thing later from Victor Dvortsov’s talk • Decide if it’s worth the effort to have a benchmark, and if so, what kind? (Off the shelf, exact replica, adjustable replica, custom-built alternate methodology, other)

19. Correct conduct of scenario analysis – CPU time • Do univariate ladder tests on all the inputs separately. Look for discontinuities, unexpected non-linearities, and especially for the Blue Screen of Death. Run scenarios out to both way above and way below any plausible values. • (If there are many inputs) Run Monte Carlo, with independent (0% correlation) inputs and extremely high volatility. The purpose is to explore as much of input/parameter space as you can, to look for pockets or edges of unexpected results. • Remember the ensemble statistics are not what you’re after. You need to be able to identify and analyze individual scenarios that stressed the model. None of these scenarios are intended to be realistic.

20. Correct conduct of scenario analysis – more CPU time • Design and run scenarios that might stress the model – various path shapes, just missing or just barely crossing any trigger levels, etc. • This should be informed by the MC in the previous step. • Test if model outputs are self-consistent. For example, does a delta or vega match what happens with a small shift?

21. Correct maintenance of scenario analysis – Historical Backtesting • Model validation is more than just backtesting, but historical backtests are an important step. • Since historical backtesting is easy to automate, and can be run without extra user inputs, best practice would be to frequently run historical backtests and have the computer flag or notify you if and when something odd shows up.

22. Limitations of Scenario Testing • All models have limitations. These are just models, not reality. • Since these tests only change inputs, and not “baked-in” assumptions, parameters, nor methodologies, a model could behave entirely self-consistently with different inputs yet still have flaws or issues. • Historical backtests where the model is confronted with reality in many scenarios, might not cover historical periods stressful enough to expose model issues. No historical scenario using a currency with a pegged FX rate can test how the model handles FX rate shifts.

23. More Limitations of Scenario Testing • Hypothetical scenarios may not discover some regions where the model has issues, especially when the extreme values are okay but the questionable area is in the middle somewhere with seemingly innocuous paths. • Although some models can be shown to have issues, you cannot in general prove that none exist; only that you didn’t find any. • Scenario testing may uncover potential issues but has nothing to say on what to do about them. You can • Avert your eyes and hope it doesn’t happen in reality, (POOR) or • Pass on the information and go away, (NOT GOOD) or better yet • See if there is a fix for the issue (PREFERRED)

24. Conclusions • Both hypothetical and historical scenarios are important • Validation scenarios need not be similar to “production” scenarios. • Stress the model until it falls off the cliff, even if the cliff seems too remote for the market to ever reach. • Agree on how your scenario test results will be used before you start any tests – buy-in is important. • Scenario testing is a useful tool to have in the validator’s toolbox • Scenario tests raise questions, but might not answer them

25. Questions?