Validating Vendor Models
Community Banks typically rely on key models that are developed and/or maintained by third-party vendors. Validating these models can be particularly challenging as compared to validating models developed in-house. Challenges typically include issues with obtaining access to the model code, detailed documentation and general lack of transparency due to “proprietary” reasons.
Regardless of these challenges, with the FDIC’s (FIL-22-2017) adoption of the Supervisory Guidance on Model Risk Management previously issued by the Board of Governors of the Federal Reserve System (“FRB”) (SR 11-7) and the Office of the Comptroller of the Currency (“OCC”) (OCC Bulletin 2011-12), the expectation is that “Vendor products should nevertheless be incorporated into a bank’s broader model risk management framework following the same principles as applied to in-house models, although the process may be somewhat modified.”
So, what to do when it comes to validating vendor models?
For smaller banks, while it is possible to meet the requirements, without in-house staff or technical, quantitative model developers, a consistent and robust process for managing vendor models is recommended to effectively manage model risk from vendor model.
Third-party vendor models, data and/or products should be incorporated into the bank’s broader model risk management framework following the same principles as applied to in-house models, although the process may be somewhat modified. Within this MRM framework, there should be clarity in the roles and responsibilities for vendor models and appropriate processes should be outlined for the process of selecting a vendor model. Vendor selection should include the proper engagement of both business and model risk resources to identity alternatives and ensure the selected model will comply with the banks MRM framework.
Documentation is key for all models and for vendor models the level of documentation provided by the vendor is likely to vary and be incomplete, it is the responsibility of the first line of defense (i.e. model owner at the bank) to generate model documentation and developmental evidence. Hence, the first line’s model documentation package should address the banks’ specific implementation of the model, as well as, the following:
- Discussion of the model’s purpose and specific application, including business and functional requirements achieved by the model
- Discussion of model theory and approach, including algorithms, calculations, formulas, functions and programming
- Description of the model’s structure
- Identification of model limitations and weaknesses
- Comprehensive list of model inputs and assumptions, including their sources
- Comprehensive list of outputs and reports and how they are used, including downstream systems that rely on them
- Description of testing (benchmarking and back-testing)
We have heard some refer to testing the model by comparing to actuals as their way of validating the model. We want to be clear, this is not validation of the model, but rather backtesting of the model. There is a difference between Backtesting and Validation. These are entirely different processes, but equally important to the overall integrity of any modeling process and the resulting usefulness of the reports.
- Back Testing is the process of reviewing the projections of a model after the fact and comparing those projections against actual performance.
- Model validation, in contrast to back testing, is completely different – validation is the structured process of “validating” the functionality and completeness of a model that involves the key guiding principal of “effective challenge” that is the critical analysis of the model. The validation process is essentially an independent review of the logical and conceptual soundness of the model.
The key to effective validation of vendor models is to address aspects associated with applicability of the model to the specific bank as outlined in the table below. By leveraging this type of framework and implementing necessary controls around model risk, community banks can favorably navigate vendor model validation challenges and benefit from their many advantages.
“Smaller banks that rely on vendor models may be able to satisfy the standards in this guidance without an in-house staff of technical, quantitative model developers. However, even if a bank relies on vendors for basic model development, the bank should still choose the models and variables that are appropriate to its size, scale, and lines of business and ensure the models are appropriate for the intended use.”, SR 11-7 / OCC 2011-12