Risk forecasting models are least reliable when needed the most

November 23, 2024

When measuring model risk with risk ratios, we find it highest when markets are in turmoil. In other words, model risk is highest, and the risk forecasting models least reliable, when they are needed the most.

My definition of model risk is:

The risk of losing substantial amounts of money when using a risk model to make decisions.

There are many ways to measure model risk. I find the risk ratios method, proposed in my paper, Model Risk of Risk Models, particularly convenient in the work I do.

It is based on a very simple idea: how much do risk forecast models disagree?

Start by using a number of different risk models to forecast risk. Then, define the risk ratio as the ratio of the highest risk forecast to the lowest.

Here is one example of what it tells us. Figure 1 below shows the highest and the lowest end-of-month risk forecast for JP Morgan from 2007 until the end of 2009. It covers the calm pre-crisis period, the global crisis in the autumn of 2008 and what happened after.

Figure 1. JP Morgan maximum and minimum forecast

The blue line show the highest and the red the lowest risk forecasts, while the labels indicate which method produced the number. Whenever a new method gives the highest or the lowest, the label changes.

Figure 2. JP Morgan model risk

We see that model risk appears very low in the calm pre-crisis period model. All the techniques mostly agree.

It all changes as we get into the crisis, with June 2007 a good starting point because that is when the quant crisis started. From that point until the crisis is mostly over, the unconditional methods, MA and HS, give the lowest forecasts. But after that, they produce the highest forecasts. Similarly, the conditional methods produce the highest forecasts during the crisis and the lowest after the crisis.

The reason for this result is obvious to anyone who knows risk models. I find similar when looking at the 2015 Swiss FX shock. We can draw at least two lessons from this.

The first is that risk models are not very good at forecasting market risk, and they are especially bad during crises.

The second is that one should not use the same risk model for all states of the world. Instead, the model should be chosen based on what is happening in the market. Do many financial institutions do that? And would the supervisors allow it?

en fin

---

---