What the AI cyber risk debate misses

The financial authorities are worried about AI cyber risk. Policy reports, speeches and financial stability reviews all warn that AI lowers the cost and raises the power of attacks on the financial system, and the volume has only grown as frontier models such as Mythos take on tasks that once required skilled human attackers. We are now seeing open-source models that seem to be nearly as capable as Mythos, without any safeguards.

The concern is justified. AI helps criminals find loopholes, lets attackers orchestrate synchronised strikes on financial infrastructure, and gives nation states plausible deniability.

The advantage runs to the attacker. A defender has to protect the whole system, every institution, every connection, every ageing legacy component, while an attacker needs only one critical weak point. Finding weak points is exactly what AI is good at. And the weak points that matter most are not individual banks but the plumbing that connects them, the payment systems, the clearing houses and the custodian banks.

But one ingredient tends to be missing from the discussion. The double coincidence.

Suppose a serious cyber attack lands on the financial system on a calm day, as it almost always would. Liquidity is ample, trust is intact, and the private and public sectors absorb the hit. The event ends up as a costly operational incident and a case study.

Now let the same attack land in the middle of a liquidity crisis, like the Covid dash for cash in March 2020 or the days after Lehman failed in September 2008. The attack and the crisis amplify each other. Institutions that would have absorbed the loss are busy protecting themselves and everybody wants cash at the same time. Payments become uncertain, liquidity is hoarded and settlement fails, deepening the very stress that made the attack dangerous.

The authorities are stretched as their attention and resources are consumed by failing markets and institutions, so a cyber attack landing at the same time competes with everything else for scarce capacity.

And it arrives just as their credibility, the main tool for calming a panic, is draining away.

This is why the same system that absorbs an attack on a calm day amplifies it in a crisis, what I called the double coincidence back in 2016.

The severity of the attack is therefore the wrong test. The state of the system when the attack arrives matters more than the attack.

Why is the double coincidence so easy to overlook? Because the data lies. Almost every cyber attack is contained. The system handles incidents all the time, and each contained attack is a data point saying that attacks of that size are manageable. The defences worked, the incident report is filed, and confidence grows. The authorities conclude they know how to handle it.

When contingency plans are informed by the record of contained incidents, they are aimed at the attack rather than the state it lands in.

The result is false confidence and preparations for the wrong event. That makes an AI crisis more likely, since the case that matters, the attack that coincides with stress, is precisely the one the preparations were not built for.

The state of the system matters as much as the size of the shock — the double coincidence problem.