On October 20, AWS reported an outage which sent the internet spiraling, here is what we know:
15:50 – Reactions start to come in
Academic experts have started to send in their responses to the outage.
Prof Jon Crowcroft FRS FREng, Marconi Professor of Communications Systems, University of Cambridge:
“One interesting challenge is that the back channels a lot of tech people use to communicate information/tech details about ongoing outages are also taken down by this outage – hence our usual ways of learning (e.g. via signal or slack) are both currently stymied by the AWS outage.”
Dr Saqib Kakvi, from the Department of Information Security at Royal Holloway, University of London:
“The issue is rooted in the DynamoDB service in Amazon’s US-EAST-1 region. The exact nature of the fault is currently not publicly available, but AWS reports they are working to repair it.The most likely mitigations are distributing the load to the three remaining US Regions or even further to the two Canadian Regions and the eight European Regions.
“Another option is to start up backup hardware in the US-EAST-1 region with a known working configuration as the faulty versions are repaired. It would be likely that full service will resume by EOD.”
Patrick Burgess of BCS, The Chartered Institute for IT:
“Given the scale of Amazon Web Services, which supports much of the world’s digital infrastructure, it’s not unusual for incidents like this to have a broad impact. Amazon tends to be transparent and proactive when resolving outages, so we can expect further updates and a swift resolution. This does, however, highlight how interconnected and reliant our everyday digital services have become on a small number of global cloud providers. Building resilience and ensuring diversity across these systems is essential to maintaining trust and continuity in our digital economy.”
Dr Junade AIi, Software Engineer, Cyber expert and Fellow at the Institution of Engineering and Technology:
“Single points of failure are a growing concern when it comes to the resilience of technical systems. This issue highlights the challenges with depending on single cloud computing regions from single cloud computing vendors and highlights the need for resilience to be built-in to essential services which people are expected to rely upon.”
15:15 – DNS fixed, APIs now the issue
Disruption appears to have be on the decline, but a number of companies have begun recording a second wave. And Amazon have an update to address:
In essence, AWS is having problems in its big US-East-1 data centre, so many control actions are failing or slow. Starting new servers there is unreliable, and Amazon is temporarily limiting how many can be created to steady the system.
Existing servers are being kept running, but some dashboards, logs (CloudTrail) and event triggers (EventBridge) may show delays. They’re fixing it zone-by-zone.
13:30 – Lloyds services back up and running
12:30 – Further update from Amazon
“We continue to work to fully restore new EC2 launches in US-EAST-1. We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically.
We are pursuing further mitigation steps to recover Lambda’s polling delays for Event Source Mappings for SQS. AWS features that depend on Lambda’s SQS polling capabilities such as Organization policy updates are also experiencing elevated processing times. We will provide an update by 5:30 AM PDT.”
11:20 – AWS outage can trigger “domino effect” to payment flows
With many banks feeling the connectivity effects of the AWS outage, Monica Eaton, Founder and CEO of Chargebacks911, believes outages like this can cause harm and even spiral out of control if not properly addressed.
She told Payment Expert: “When AWS sneezes, half the internet catches the flu. Outages like this cause frustrated users, but also triggers a domino effect across payment flows. Failed authorisations, duplicate charges, broken confirmation pages, all of that fuels a wave of disputes that merchants will be cleaning up for weeks. And once a customer files a dispute, you are already on the back foot.
“What I expect now is a spike in ‘I never got my service’ or ‘I was charged twice’ claims. Many of those won’t be fraud, just confusion. But confusion is the number one driver of chargebacks. If merchants sit back and wait for disputes to roll in, they will bleed revenue unnecessarily.
“The smart move is to get ahead of the narrative. Run duplicate charge sweeps. Push proactive notifications to affected users. Document the outage window for clean evidence. Offer fast refunds where appropriate. It is cheaper to fix misunderstandings than fight losing battles in the dispute process.
“The outage will end long before the disputes do. Any business that treats this as a one-day incident is already behind. Downtime happens, but silence and slow responses are what cause real damage.”
11:15 – Square back up and running
A large share of the internet’s payment systems runs on Square which was also knocked out by AWS outages. Now it’s status has changed following an internal fix being deployed.
11:00 – Starting to normalise
Amazon outage reports are down below 1,000, from their peak of around 15,000+ earlier today.
And UK banks and government services are also reporting almost normal levels of disruption.
10:30 – From disruption to downgraded
Amazon says the initial DNS fix is live and showing early signs of recovery. Work isn’t finished, though: expect elevated latency and some bumps while they work through a sizable backlog. In short, the fix is rolling out, and AWS is monitoring it closely.
10:20 – Halifax and Bank of Scotland acknowledges AWS outage is “impacting some of our services”.
10:11 – Coinbase says ‘all funds are safe’
10:10 – Update from Amazon
“Oct 20 2:01 AM PDT We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery. This issue also affects other AWS Services in the US-EAST-1 Region. Global services or features that rely on US-EAST-1 endpoints such as IAM updates and DynamoDB Global tables may also be experiencing issues. During this time, customers may be unable to create or update Support Cases. We recommend customers continue to retry any failed requests. We will continue to provide updates as we have more information to share, or by 2:45 AM.”
10:05 – HMRC hit
Concerningly, it looks like UK government services are also being hit.
10:00 – Difference between UK/US infrastructure
In the US, there are now signs the peaks of the outage match up quite nicely with AWS’ movement signalling the development teams may have identified the issue.
However, in the UK, it is a different story. While in the US it’s looking like all private companies have been hit by the AWS issues, in the UK the issue is a little more deeply rooted in some key infrastructure. Online banking is down.
09:50 – A fix may have been implemented
Peaks of outage reports are starting to fall off. All we know at the moment is that Amazon is working on the issue.
09:45 – What’s being done now
AWS has engaged response teams and is posting rolling updates as it works through the incident; initial public notices landed earlier this morning and confirm the US-EAST-1 dependency. Expect intermittent recovery before full resolution; keep an eye on AWS Health updates and vendor status pages.
09:35 – Ripple effects hit big apps and finance
Outage trackers and company updates point to widespread disruption across Amazon/Alexa, Snapchat, Fortnite/Roblox, Ring, Canva, Duolingo and more.
In financial services, Coinbase has acknowledged impact and says funds are safe; lists circulating also include Robinhood, Venmo and Chime among affected services. UK users are reporting banking login problems at Lloyds, Halifax and Bank of Scotland amid the wider AWS issues.
09:20 – AWS confirms major incident in US-EAST-1
Amazon Web Services says it is seeing “increased error rates and latencies for multiple AWS services” in the North Virginia region, with case creation via Support Center also affected. Engineers are working to mitigate the issue and determine root cause.
