Download
1 / 8
E N D
CompatibL Model Validation About CompatibL CompatibL Technologies was founded in 2003 to offer risk technology solutions to banks and asset managers. Since then CompatibL has established itself as the leading provider of quantitative software and services for derivatives analytics, limits, and regulatory capital. CompatibL has over 200 experienced developers and financial engineers and boasts a client base of over 50 banks, central banks, supranationals and asset managers in the US, EMEA and Asia, including 4 out of 5 largest derivatives dealers. Over 140 major projects have been implemented across this client base. CompatibL is headquartered in Princeton (United States) with the main delivery centres located in London (UK), Warsaw (Poland) and Singapore. A new approach to model risk control www.compatibl.com info@compatibl.com Princeton (USA) CompatibL Technologies LLC 100 Overlook Center Second Floor Princeton, NJ 08540, USA London CompatibL Technologies Ltd First Floor 100 Pall Mall London, SW1Y 5NQ, UK +1 (609) 919 0939 +44 (20) 3743 8800 Singapore CompatibL Pte. Ltd. 30 Cecil Street 19-08 Prudential Tower Singapore, 049712, Singapore Warsaw CompatibL Sp. z o.o. Prosta 32 Second Floor Warsaw, 00-838, Poland +65 6813 2067 +48 (22) 110 8005
About CompatibL Model Validation CompatibL is a leading software vendor and consultancy that combines advanced application engineering with top quant expertise for the capital markets industry. The company provides enterprise risk solutions and multi-asset analytics for front, middle and back office. CompatibL is also a leading provider of custom software development services and model validation consultancy for the financial industry. CompatibL offers independent validation services across a wide range of valuation and risk models, including formal regulatory validation reports as well internal reviews with actionable insights to mitigate models’ weaknesses and limitations. We have expertise in the latest quantitative research and model validation methodologies as well as implementation practices. Our software is running in over 85 financial institutions worldwide, including 4 of the 5 largest dealers, 33 central banks and some of the world’s largest asset managers in the Americas, EMEA and APAC. Over the years, our products have won industry recognition for their quality, performance and flexibility. CompatibL has teams located in strategic global locations across Europe, North America, Asia working in close contact with clients. We also have experience in long-term partnerships and have been working closely with the clients’ internal teams for many years. 15+ 200+ years experience in software development for financial institutions in-house experts, including quants, software engineers, solution architects, UI/UX designers and analysts in 4 branch offices 140 85 Market Risk Technology Vendor of the Year Award CompatibL Risk Product of the Year Award Alexander Sokol Quant of the Year Award delivered enterprise software projects and hundreds of quantitative consultancy and model validation engagements clients worldwide, including 4 of the 5 largest dealers, 33 central banks and some of the world’s largest asset managers 2 / CompatibL Model Validation CompatibL Model Validation / 3
Why Choose CompatibL for Model Validation? Technology behind the Validation Today, there is significant emphasis on model risk management and validation. A transparent and independent model review process is an integral part of modern model vetting and model risk frameworks. Our validation approach is based on the CompatibL Model Validation Platform, which integrates into a single environment all the code necessary to specify, automate, run and report model tests for third party and proprietary models. It can be used not only to produce a formal model validation report, but also as a benchmark or challenger model to assess the accuracy of the primary model. As a vendor of market and credit risk software, CompatibL has hands-on experience and expert judgement in all aspects of model QA and review processes. Over the years, we validated hundreds of models and software solutions for banks and asset managers across multiple model types with applications to valuation and risk management. This knowledge and experience enabled us to develop an innovative, rigorous and cost-effective approach to model validation. CompatibL Model Validation Platform is focused on model validation, rather than model development, and provides a wide range of functionality out-of-the-box, saving the validation team the time and expense of creating these components during the project. It supports a variety of model input formats, including those of mainstream vendors and provides a set of common statistical acceptance tests, instrumentation for running regression tests, and a comprehensive set of tools to produce reports on individual test cases and consoli- date them into a model validation report template for internal or regulatory use. Advantages of Model Validation with CompatibL A Wide Variety of Models Highly qualified expert team Benefit from access to a wide range of quantitative and tech- nical resources and ensure that top experts work on validating models critical to your business. Independence Rely on independent high-quality model validation performed by a team with extensive experience and a solid model validation code framework. Cost savings Our approach reduces the costs compared to a traditional vali- dation approach as we reuse standard validation code compo- nents gathered over time in our model validation platform. The platform provides a rich spectrum of single-asset and multi-asset hybrid models to assess the impact of model choice on prices and risk measures, providing an additional dimension to model risk assessment. CompatibL Model Validation Platform provides an open API and can be customized, permitting non-standard models and alternative benchmarks to be used. Benchmarking across a range of market practice models is encouraged by many supervisory guidelines. It also helps to reduce the risk of implementation error and prevent mis-specification of the model. Documentation The documentation delivered during the validation process is comprehensive and sufficient to independently reproduce the calculation results. Experience with in-house and vendor models Deep analysis of model assumptions, limitations and output CompatibL has experience with the leading quantitative model types published in academic literature, and with the imple- mentation of these models by software vendors, leading to more effective and stream- lined validation process. We don’t stop at producing a model validation report and also help you to take a wider view on model risk, its origins and potential impact on your business. 4 / CompatibL Model Validation CompatibL Model Validation / 5
Model Coverage Models supported by CompatibL Model Validation Platform out of the box: Our expertise spans real-world and risk-neutral multi-factor and multi-asset risk models for most asset classes including interest rates, foreign exchange, equity, inflation, credit and commodities. Multi-curve real world and risk neutral valuation, capital and PFE models Capital and regulatory metrics: - Basel II, II.5, III, early IV - IRRBB - SA-CCR - Regulatory MPOR - ISDA SIMM - FRTB for market and credit risk, including the latest revisions and consultative documents Advanced multi-currency CSA and collateral simulations All major risk metrics and valuation adjustments, including IM and MVA Capital optimization models Global regulatory standard coverage Operational Risk Basel II Basel IV Market Risk FRTB IRRBB Basel II.5 Credit Risk SA-CCR Basel III Systemic Risk ISDA SIMM Margin 2006 2008 2010 2012 2014 2016 2018 2020 Timeline 6 / CompatibL Model Validation CompatibL Model Validation / 7
Structured Approach Benefits of the Structured Approach When dealing with complex models, we first identify key components in their calculation flow and consider them as independent unit models, each subject to rigorous individual testing. For each component we follow the validation methodology and steps shown below. Once each component has undergone independent testing and passed its strict acceptance criteria, we connect the components and run consolidated tests. We then verify the integrity of the model overall and ensure that all input and output data are properly connected in the full model. Once we have ensured that the calculation flow works as expected, we perform tests and analyze the entire model. Traditional approach Monolith model Inputs Outputs Monolith model Inputs Outputs Validation Focus not only on the entire model but also on its individual components Unit tests for model components lead to easier to understand, more reliable test criteria and implementation CompatibL approach Composite Model Validation Validation Composite Model Validation Validation Components Components Components Components Better isolation of model weaknesses and identification of error sources More clarity to remediation actions, reducing the risk of delays to model use in production Components Components Inputs Outputs Components Components Validation Validation Validation Validation Consolidation, Validation Consolidation, Validation 8 / CompatibL Model Validation CompatibL Model Validation / 9
Automation Validation Steps Automation and open API of CompatibL Model Validation Platform provide the ability to continuously retest the model as improvements are made and remediation action is taken. Following validation project completion, it can be rerun for model updates and new releases without CompatibL’s assistance. This enables continuous assessment of the model risk on an ongoing basis with low recurring costs. CompatibL’s approach to model validation and verification includes the following stages for a comprehensive and detailed review of the model and its components. Model specification stage: • Analyze model documentation, verify its consistency, level of detail and correctness • Analyze the relevant regulatory guidance and market practice, model choice applicability and adequacy, and propose operationally feasible alternatives • Identify and assess key model inputs and outputs, verify their completeness and proper use of market data for a given model • Identify and assess key model assumptions and limitations, then compare to other model alternatives Validation specification stage: • Identify and assess key validation assumptions and limitations, given input and output data and possible operational constraints • Review data scope, validation criteria and functional tests for model vetting based on the results from the previous stage Testing stage: • Run validation tests • Assess the impact of model choice on material measures Compared to the traditional approach of using manual tests on hardcoded inputs and direct integration with market data, our model replication approach enables a wider validation scope, not limited by simple cases and analytical examples. If desired, it can be scaled to a full live portfolio, providing ultimate confidence in the validation results. Conclusion stage: • Analyze individual test results, provide recommendations and conclusions • Gather all findings into an executive summary and a highly detailed full validation report, conforming to the relevant regulatory validation template when applicable Overall, our approach to the automation of model validation tests provides depth and scale advantages over the legacy validation methodologies. Delivery stage: • Deliver validation documentation, reports, and supporting materials • Deliver test code artifacts and detailed documentation that allows this validation to be rerun in-house after project completion 10 / CompatibL Model Validation CompatibL Model Validation / 11
Test Types The first category of tests includes tests that verify theoretical expectations and asymptotic conditions. This category is the starting point of any model validation process, covering the most basic form of validation. However, with advanced models such tests are insufficient. Overconfidence in validation results obtained only from this first category of tests can lead to material errors due to operational or functional issues in more complex real-world cases. The validation process followed by CompatibL’s team embraces both categories of tests. CompatibL Model Validation Platform provides base models and statistics functionality for the first category of tests, and many out-of-the-box components for use by a replicating model for the second category of tests. be run as part of the validation project deliverables to provide assurance of validation code accuracy. We include standard algorithms, such as American Monte Carlo (AMC) and hybrid model calibration, which are applicable across all asset classes and risk factors and enable a wider portfolio scope. The platform provides lightweight and transparent API that reduces the implementation costs. The second category includes independent replication of the model and its components from the first principles according to the model specification. Such replication should give identical or statistically close results, provided the same input data is used in both production and challenger environments. The components are verified by a com- prehensive set of internal tests, which can Quantitative Test Approach Challenger Platform Approach Models Analytic Semi-analytic Asymptotic Approximate Model Actual vs Expected Models Independently replicated Alternative Model Actual vs Expected Actual model Actual outputs 1 Conclusion Conclusion Test case 1: Inputs 1 Expected model 1 Expected outputs 1 Test case 1: Inputs 1 Actual model Consolidated actual outputs Consolidated inputs Expert judgement + Expert judgement Challenger open-source model Consolidated expected outputs Test case N: Actual model Actual outputs N Inputs N Test case N: Inputs N Expected model N Expected outputs N 12 / CompatibL Model Validation CompatibL Model Validation / 13
Transparency and Openness Model Documentation Unlike most model validation consultancies, CompatibL provides the source code for all model validation tests, including the code in the CompatibL Model Validation Platform. This gives you additional confidence in validation results, and further reduces operational and model risk. Because complete and accurate model documentation is critical to model risk management, we include the review of code structure and quality as part of the validation project. We ensure that documentation is sufficient for users and decision makers to have complete confidence in how the model operates, including its key inputs, outputs, assumptions and limitations. When improvements are needed, we deliver enhanced documentation with the validation report. How we do it? We verify that the existing model documentation meets the regulatory guidelines. We provide recommendations in case of any documentation gaps or inconsistencies. We deliver enhanced documentation with the validation report. Professional presentation of the report We use the professional LaTeX typesetting system used by most financial book publishers to provide test result presentation within the report. We automate the generation of test case documents to produce professional supporting documentation. CompatibL contributes to the model validation community and supports ModVal.org, a free online resource for regulatory and internal model validation. ModVal.org provides detailed calculation walkthroughs and validation spreadsheets for some of the most complex regulatory calculations: In our test-case templates and methodology review, we follow a unified document structure, which covers the following items. Model review: Validation model review: Analysis and summary: Internal Model Approach for market risk (IMA-TB) Standardised Approach for market risk (SA-TB) Standardised Approach for CVA (SA-CVA) Basic Approach for CVA (BA-CVA - Background, market practice and methodology - Detailed test descriptions and complete inventory of the required input data - Detailed analysis of test results, providing the foundation for report conclusions - A thorough analysis of model specifications, assumptions and limitations - Detailed instructions on how to perform tests and independently reproduce their results - Model risk assessment and conclusions CompatibL has developed and presented several training courses to help model validation specialists to broaden their expertise in the latest regulatory frameworks. Our experts can customize the training program to suit the needs of your quants and model validation experts. 14 / CompatibL Model Validation CompatibL Model Validation / 15
