OCC Guidance Focuses on the Risks of Using Computer-based Financial Models

WASHINGTON — The Office of the Comptroller of the Currency issued guidance today stressing the importance of regularly validating the computer-based financial models that are used to help with decision-making on a range of important activities at banks.

In its guidance, the OCC said that over the next decade these models will increasingly guide banks' enterprise-wide risk management, economic and regulatory capital allocation, whole-bank credit risk, fiduciary asset management, and internal profitability measurement.

Computer-based financial models are now routinely used for credit scoring, asset-liability management, trading-risk management, and for the valuation estimates of financial instruments, such as retained interests in securitizations.

"This guidance applies to small banks that use vendors to supply models just as much as to the largest banks," said Jeffrey A. Brown, director of OCC's Risk Analysis Division. "Virtually all banks use models today. It has become part of the fabric of banking."

While Mr. Brown's specialized unit of Ph.D.-level economists examine some of the most complex models used by most large banks and some smaller banks, a look at banks' model validation and testing methods has become a part of all OCC bank examinations.

While bank board members are not expected to understand the complex formulas upon which the models are built, they are expected to receive plain-English explanations of what the models do. "These models are simply tools used to help decision makers solve business problems," said Mr. Brown.

In the bulletin, the OCC cautioned that it "has observed several instances in which decision makers either relied on erroneous price or exposure estimates, or on an overly broad interpretation of model results, with serious consequences for their bank's reputation and profitability. There are many more instances in which the incorrect use of models created the potential for large losses, which were avoided only fortuitously."

The OCC said that errors can occur in any of the three components of a model: the information input component, which delivers assumptions and data to the model; the processing component, which contains the theoretical model and transforms inputs into estimates using computer programs; and the reporting component, which translates the mathematical estimates into useful business information.

Model validation, said the OCC, requires considerable subjective business judgement as well as technical expertise. The general procedures that apply to model validation include: (a) independent review of the logical and conceptual soundness, (b) comparison against other models, and comparison of model predictions against subsequent real-world events.

While model validation can be costly, particularly for smaller banks, the OCC said, "using unvalidated models to manage risks to the bank is potentially an unsafe and unsound practice."

In its bulletin, the OCC said it expects that formal policies ensure the following goals are met:

• Decision makers understand the meaning and limitations of a model's results.
• When a model has been in use for a reasonable period of time, its results are tested against actual outcomes.
• The bank should demonstrate a reasonable effort to audit the information inputs to the model and address any errors in a timely fashion.
• The seniority of the management overseeing the modeling process should be commensurate with the level of risk.
• Model validation must be independent from model construction when feasible.
• Responsibilities for the various elements of the model-validation process must be clearly defined.
• Modeling software should be subject to procedures that allow developers and users to have the ability to change code only upon review and approval by an independent party.

The Risk Modeling bulletin (OCC 2000-16) is posted on the OCC's Website: www.occ.treas.gov.