Validating Vendor Models

Community Banks typically rely on key models that are developed and/or maintained by third-party vendors. Validating these models can be particularly challenging as compared to validating models developed in-house. Challenges typically include issues with obtaining access to the model code, detailed documentation and general lack of transparency due to “proprietary” reasons.

Regardless of these challenges, with the FDIC’s (FIL-22-2017[1]) adoption of the Supervisory Guidance on Model Risk Management previously issued by the Board of Governors of the Federal Reserve System (“FRB”) (SR 11-7)[2] and the Office of the Comptroller of the Currency (“OCC”) (OCC Bulletin 2011-12)[2], the expectation is that “Vendor products should nevertheless be incorporated into a bank’s broader model risk management framework following the same principles as applied to in-house models, although the process may be somewhat modified.”

So, what to do when it comes to validating vendor models?

For smaller banks, while it is possible to meet the requirements, without in-house staff or technical, quantitative model developers, a consistent and robust process for managing vendor models is recommended to effectively manage model risk from vendor model.

Third-party vendor models, data and/or products should be incorporated into the bank’s broader model risk management framework following the same principles as applied to in-house models, although the process may be somewhat modified. Within this MRM framework, there should be clarity in the roles and responsibilities for vendor models and appropriate processes should be outlined for the process of selecting a vendor model. Vendor selection should include the proper engagement of both business and model risk resources to identity alternatives and ensure the selected model will comply with the banks MRM framework.

Documentation is key for all models and for vendor models the level of documentation provided by the vendor is likely to vary and be incomplete, it is the responsibility of the first line of defense (i.e. model owner at the bank) to generate model documentation and developmental evidence. Hence, the first line’s model documentation package should address the banks’ specific implementation of the model, as well as, the following:

  • Discussion of the model’s purpose and specific application, including business and functional requirements achieved by the model
  • Discussion of model theory and approach, including algorithms, calculations, formulas, functions and programming
  • Description of the model’s structure
  • Identification of model limitations and weaknesses
  • Comprehensive list of model inputs and assumptions, including their sources
  • Comprehensive list of outputs and reports and how they are used, including downstream systems that rely on them
  • Description of testing (benchmarking and back-testing)

We have heard some refer to testing the model by comparing to actuals as their way of validating the model. We want to be clear, this is not validation of the model, but rather backtesting of the model. There is a difference between Backtesting and Validation. These are entirely different processes, but equally important to the overall integrity of any modeling process and the resulting usefulness of the reports.

 

  • Back Testing is the process of reviewing the projections of a model after the fact and comparing those projections against actual performance.

 

  • Model validation, in contrast to back testing, is completely different – validation is the structured process of “validating” the functionality and completeness of a model that involves the key guiding principal of “effective challenge” that is the critical analysis of the model. The validation process is essentially an independent review of the logical and conceptual soundness of the model.

The key to effective validation of vendor models is to address aspects associated with applicability of the model to the specific bank as outlined in the table below. By leveraging this type of framework and implementing necessary controls around model risk, community banks can favorably navigate vendor model validation challenges and benefit from their many advantages.

 

“Smaller banks that rely on vendor models may be able to satisfy the standards in this guidance without an in-house staff of technical, quantitative model developers. However, even if a bank relies on vendors for basic model development, the bank should still choose the models and variables that are appropriate to its size, scale, and lines of business and ensure the models are appropriate for the intended use.”, SR 11-7 / OCC 2011-12

 

To learn more, download the HnC Smart Solutions white paper on Validating Vendor Models for Community Banks.

Validating Vendor Models White Paper

 

“Community banks with assets greater than $1 billion have now joined the other larger banks in the regulatory expectations around formal management of model risk”

Considerations for Model Risk Management for Community Banks

With the adoption of Supervisory Guidance on Model Risk Management by the FDIC, (FIL-22-2017) community banks with assets greater than $1 billion have now joined the other larger banks in the regulatory expectations around formal management of model risk.

This is not surprising, considering the ever-increasing reliance on models for decision making and the many high-profile cases of model failure, such as LTCM and the others in the run up to the financial crisis, led to significant losses and reputational damage for financial institutions.

As community banks address this new requirement, below are some points for consideration:

  1. Model Definition – The regulatory guidance defines the term model as “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates”. Further “the definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgment, provided that the output is quantitative in nature”.  As this is a broad definition, defining what is a model can be a potential source of many varying views and disagreements.  It will be important for all stakeholders to have a consistent view of what constitutes a model so that all potential models can be identified. Examples of models include, Vendor models such as FICO scores, custom acquisition, account management and recovery scorecards for retail and wholesale, loan pricing, expected loss models (ALLL), Asset Liability Management (ALM) models and unexpected loss models (i.e., economic capital, regulatory capital, stress testing).
  2. What is model risk – Model risk can arise when adverse consequences—such as poor business and strategic decisions, financial losses, regulatory or legal penalties, or damage to a bank’s reputation—result from the use of a model whose data, assumptions, design, underlying theory, output, or control environment are not appropriate.
  3. Key components of a model – the definition of a model can be broken down into essentially the three components. If we look at these components of a model, we can start to assess some of the sources of potential model risk. Errors can occur at any point from design through implementation.  Making assumptions is unavoidable when building models, so the challenge is in making assumptions which do not render the model useless for its intended purposes
  4. Regulatory requirements for managing model risk:Model risk should be managed like other bank risks (e.g., credit and market risk), and managed throughout the model life cycle. It requires broadened responsibilities of various individuals in the bank and stepped-up efforts regarding model governance and model approval, model documentation and testing, ongoing performance monitoring, and proper management of model changes
  5. The key components of a Model Risk Management (MRM) program:These include:
    1. Strong Model Governance, Policies & Procedures
    2. Clear roles and responsibilities,
    3. Process for model initiation & identification and risk tiering,
    4. A comprehensive inventory of models,
    5. Well defined processes and procedure for model development and model validation and appropriate model approvals
    6. Model Testing, and implementation
    7. Model documentation
    8. Independent review resulting in model ratings, and model risk issues management
    9. Ongoing model management,
    10. Model performance monitoring
    11. Model risk assessment and reporting
    12. Use of vendor and other third-party models should be incorporated into the model risk management framework

In establishing and operationalizing an effective model risk management framework, community banks have several choices to make. What is important is to get the right balance between sophistication vs. practicality that is commensurate with the bank’s business model.

The importance of this effort cannot be understated as the model risk management framework will need to stand the test of regulatory scrutiny and safeguard the bank from potential future losses and reputational damage.

To learn more, download the HnC Smart Solutions white paper on Model Risk Management for Community Banks.

Download MRM White Paper