Misleading measurements and garbage in - garbage out models of systemic risk

May 23, 2021

There are many measurements of systemic risk and a huge number of papers depend on those measurements. But is that work of any use? There is an archetypical story from May 1944 where the staff of General Eisenhower called up the Army's meteorological office to ask for the weather forecast for Normandy on June 6. The meteorologists responded by saying, “Any weather forecast a month in the future is meaningless.” The response from the general's staff was: “Yes, we know that, but we need a number for planning purposes.”

In today's macroprudential world, that number is systemic risk.

Financial risk is measured by a device I like to call the riskometer, that when plunged deep into the bowels of Wall Street, pops out a measurement of financial risk. Some riskometers are meant to capture systemic risk, the likelihood of a serious dysfunction in the financial system, crisis or otherwise, like SRISK, COVaR and ECB's CISS.

To go on a philosophical tangent, it is best to distinguish between systemic risk riskometers that measure risk that is already being realized from those aiming to forecast systemic risk years into the future.

There are only two reliable indicators of future systemic risk, credit growth as in Schularick and Taylor's studies and our low risk crisis predictor. All the others only capture stress after the fact, suffer from excessive model risk, and even worse, only capture short-term fluctuations, not long-term systemic risk.

To come back to the archetypical story at the beginning of the chapter, there's a huge number of papers and policy analysis that depend on measurements of systemic risk. When these take the likes of SRISK, COVaR and CISS as correct measurements of systemic risk, a bit like a thermometer accurately measures temperature, the authors make exactly the same mistake as the staff of General Eisenhower.

Even if you don't agree, at the least acknowledge that all the systemic risk indicators disagree with easy other (easily verified) and recognise that model risk in your work. Not doing so, taking a systemic measurement as truth like so many paper do, is intellectually dishonest.

To save money and time, just use a random number generator:

SystemicRisk = rand()

As the computer scientists like to say: GIGO — garbage in - garbage out.

---

---