“Whatever can fail will fail. And it’ll fail spectacularly.”
For Spencer Kimball, Co-founder and CEO of Cockroach Labs, this is more than a warning about the unpredictability of technology. It’s the principle that should shape how financial institutions design the infrastructure supporting their most critical services.
With banks and payment providers increasingly expected to deliver uninterrupted, real-time services, the consequences of infrastructure failure extend far beyond technical inconvenience.
Speaking to Form3 Chief Revenue Officer Mark Fieldhouse on the Payments Cannot Fail podcast, Kimball argued that operational resilience has to be designed in from the start, particularly as artificial intelligence introduces new levels of automation and transaction activity.
For Kimball, whose experience includes a decade at Google, failure at scale is inevitable, so the aim is to stop individual failures from becoming business-wide disruptions.
Resilience cannot be an afterthought
Financial institutions have historically invested heavily in disaster recovery, building contingency arrangements intended to restore services following outages.
However, Kimball believes this approach can create unnecessary complexity when resilience is added to systems that were never designed to accommodate it.
He noted: “Everyone that’s ever gone down these paths finds that without starting from the right first principles at the lowest level of the system, everything else becomes a layer of kludges.
“They add up, and they have a cumulative and very unpleasant cost.”
This matters most at the database level, where information behind payment processing, account management and transaction records must stay accurate and accessible.
CockroachDB is a distributed system that replicates information across multiple locations, so it stays available when individual components fail.
Kimball explained that the effectiveness of such arrangements depends on how organisations distribute their infrastructure.
“Your weakest resilience in that stack becomes the weak link in the chain,” he stated. “And so it’s certainly not just the database, but I think the lower down in this stack, your resilience fails, the worse the ultimate problem is.”
Replicating information across multiple machines provides limited protection if those machines share the same physical rack, power supply or data centre. For financial institutions, genuine resilience may therefore require distributing infrastructure across geographical regions, facilities or cloud providers.
Kimball asked: “What does this use case require? What do the regulators require? How much do we want to spend on it? How much is the network connectivity? What are the latencies?”
Moving beyond disaster recovery
For payment providers, the difference between restoring a failed service and maintaining operations during disruption can be significant.
Kimball believes distributed database technology is changing what financial institutions should reasonably expect from their infrastructure.
He stated: “So it’s not disaster recovery, it’s operational resilience.”
Under a traditional recovery model, an outage may require systems to be restored before transactions can resume. Distributed infrastructure, however, can allow services to continue operating despite the loss of individual facilities or cloud regions, provided sufficient replicas remain available.
He added: “And what that means is that you might have a couple seconds of latency with Cockroach if a whole facility disappears, or AWS loses a region or GCP loses a region.”
He cited a major banking customer operating CockroachDB across five US data centres, combining public and private infrastructure, as an example of how institutions can distribute their operational risk.
Cyberattacks, corrupted information and incorrect data entries can still compromise systems, making independent backups and additional safeguards essential.
Kimball described this approach as “defense in depth”, arguing that resilience must extend throughout the technology stack rather than depend on a single protective mechanism.
“Having a very resilient database solves a lot of problems for you, but you want to make sure that there’s other resilience mechanisms,” he added.
The discussion also explored the origins of Cockroach Labs, the challenges of global data sovereignty and how distributed databases could operate in space. Kimball also shared his predictions for AI-driven infrastructure, rising computing demands and the challenges facing financial institutions over the coming years.
Watch the full episode of Payments Cannot Fail in the video above to hear Spencer Kimball and Mark Fieldhouse explore these topics and more.
