If one cloud region can take you down, you were never redundant

3.4 million people in Ireland couldn’t get their money

Early on Tuesday 20 October a power failure in one of Google Cloud’s London zones, europe-west2-b, knocked Revolut over. Card payments, ATM withdrawals, transfers and even logging in failed for customers in Ireland, the UK, France, Spain, Germany, Romania and Luxembourg (Irish Times, RTÉ). Google said most of its services were back shortly after midday, roughly six hours in. Revolut’s own status page was still listing 31 services as partially unavailable that afternoon (The Journal). Ireland’s other banks reported normal service all day.

My son was one of the 3.4 million. He went to buy lunch on campus; his card was declined, and he had no way of knowing whether the bank was broken or his account was. He went hungry to his afternoon lectures, which is about the mildest outcome available. Across seven countries, people were left at tills with a queue behind them, at fuel pumps, at ATMs, with bills bouncing, and with nothing to go on but a status page.

Revolut pointed at its cloud provider. That is accurate, and it is irrelevant to anyone standing at a checkout. Nobody cares whose data centre lost power. The job of a bank is to hand people their own money when they ask for it, and for most of a working day one didn’t.

Regions fail. All of them. Regularly.

Tuesday was not a Google problem; it was a cloud problem. In the last 18 months each of the big three has had a headline failure measured in hours: Google Cloud on 12 June 2025 (more than seven hours, 76 services), AWS us-east-1 on 20 October 2025 (around 15 hours, a DNS race condition in DynamoDB) and Azure Front Door on 29 October 2025 (around eight hours, a bad global config) (Channelweb, The Register). In May this year, a thermal event in a single AWS availability zone was reportedly enough to push up error rates across us-east-1 again (Tech Insider).

That last detail is the one to dwell on. The textbook advice is “spread across three zones in one region and you are covered”. Tuesday showed the limit of the textbook. Google reported a power event in one zone, europe-west2-b; the knock-on was intermittent network failure across the whole London region. Zones share power infrastructure, network fabric, control planes and, with depressing regularity, a post-incident report explaining that a single-zone event had wider impact than designed. Redundancy inside a region protects you from a server dying. It does not protect you from the region having a bad morning.

The arithmetic does not help either. A 99.99% regional SLA permits about 53 minutes of downtime a year. Revolut’s customers got most of a working day, and the SLA credit, when it comes, is a percentage of the cloud bill, not of anyone’s unpaid bills.

Regulators have noticed. EU regulators last year designated Google Cloud among 19 critical third-party ICT providers for the finance sector, and Tuesday reopened the argument about how much of the banking system now sits on a handful of them (RTÉ). If you run charging, payments, fraud control or anything a regulator calls critical, the first question after an outage is no longer “what went wrong at your provider?” It is “why did one region matter to you?”

“Geo-redundant” is not a copy in another region

Most things sold as geo-redundancy are disaster recovery with better marketing: a copy of the data in a second region, a replica that lags by a few seconds, a runbook, and somebody whose pager goes off at 06:10 to promote it. That works, in the sense that the business survives. It does not keep the service up, and keeping the service up is the only thing the customer at the till cares about. Revolut’s extra hours after Google had recovered are the tell. If recovery is a procedure, it is measured in hours.

The real thing looks different:

• Two, preferably three, full clusters in separate regions, each taking live production traffic all the time. Not a primary and a spare. Every region accepts writes, and every region can answer for any customer, account, or subscriber.
• Replication in every direction, continuously, with a defined rule for what happens when two regions change the same record at the same moment. Without that rule, you get silent divergence, which is worse than an outage because nobody notices.
• Failure absorbed by the architecture, not handled by a person. When a region drops, the others are already serving. The only thing that changes is that you have less headroom until it comes back.
• Reconciliation on reconnect. Regions keep working through a network split and catch each other up when the link returns, without an operator deciding which side was right.

A one-line test: if you pulled the plug on your busiest region at 06:00 on a Tuesday, who would have to do anything? If the answer is anyone, you have a disaster-recovery plan. That is not nothing, but it is not geo-redundancy.

The reason most platforms stop at active-passive is that active-active is hard. Accepting writes in two places at once means conflicts, and resolving them correctly, at speed, without dropping transactions, is where most general-purpose data stores give up and tell you to shard by geography instead. Sharding by geography does not remove the problem. It just chooses in advance which customers get stranded when a region fails.

How Volt does it: every region live, all the time

Volt has two layers of redundancy, and the distinction is the whole point of this post.

Inside a cluster, K-safety keeps multiple synchronous copies of every partition on different nodes. Lose a node, a rack, or a zone if you have spread the cluster across them, and the cluster carries on with no lost transactions and no intervention. That is local redundancy. Every serious platform has a version of it, and it is exactly the layer Tuesday went straight through.

Across regions, Active(N) cross-datacentre replication (XDCR) runs two or more complete clusters as peers. Every cluster accepts writes. Each partition streams a binary log of its committed transactions to every other cluster in parallel, and the receiving cluster applies the results rather than re-running the work, so replication keeps pace with the transaction rate. The clusters can sit in different regions, different cloud providers, your own data centres, or a mix; the documentation is quite relaxed about them being on separate continents.

What that buys you on a morning like Tuesday:

• No failover. There is no primary to promote and no standby to warm up. The surviving regions were already taking traffic; they simply take more of it.
• No runbook. The only action is routing: point the clients, or the load balancer in front of them, at the regions that are up. Nothing on the data side waits for a person.
• Nothing committed is lost. If a region vanishes mid-replication, the transactions it had committed but not yet shipped sit in its own command log and are delivered when it returns. The other regions carry on in the meantime.
• Defined behaviour on conflict. If two regions update the same record at the same moment, Volt resolves it by a predictable rule, timestamp-based by default or logic your application defines, and records the conflict so it can be audited. No silent divergence.
• Partition tolerance. A region cut off by a network split keeps serving locally and catches up with its peers when the link comes back.

And any region can decide for any customer. Volt does not home accounts to a region or route transactions back to a designated primary, so a subscriber normally served from London gets the same authoritative answer from Frankfurt, at local latency, without a cross-region lookup (Active(N) solution brief).

This is not a lab feature. Volt’s telco customers have run charging and policy control this way in production for years, for the simple reason that a charging system that stops is a network that stops billing, and nobody signs up for that.

One honest caveat. A geo-redundant data layer does not rescue an application tier that lives in a single region. The whole path from the card terminal to the decision has to be reachable from more than one place. Volt makes the hardest part of that, the data, a solved problem. The rest is routing and discipline.

Table stakes, not a premium tier

If a system handles money, charging, fraud decisions or network control, “a whole region is unavailable” is an expected event, not an edge case. It belongs in the requirements as a line item, not in the lessons-learned section. Yet it is still routinely scoped, priced and sold as the upgrade, the thing you add in phase two once the budget recovers. Phase two arrives at 06:10 on a Tuesday.

Three questions to put to whoever runs your data layer, or sells it to you:

1. Can two regions accept writes for the same customer at the same time, and what is the rule when they disagree?
2. In the first sixty seconds after a region drops, who has to do something? Name the person.
3. When did you last deliberately take a region down in production, and what did customers see?

Volt’s Active(N) exists so that the answers can be “yes”, “nobody” and “last month, and they saw nothing”. If a region going dark ought to be a non-event for your system, talk to us.