I've never met anyone who actually chose to use the AWS US-EAST-1 region, but almost everyone seems to have something running there. It's a kind of default, the place some of your services ended up when you decided to give this fancy “cloud” thing a try when it was a hot new idea. US-EAST-1 is, after all, the ur-AWS, the original location from where the hyperscaler hatched.
Not changing the default is a choice too, though, and yesterday a whole lot of managers came very close to having to justify that choice to some extremely angry bosses when US-EAST-1 went down, again.
Now, you know and I know that 100% uptime is a fairy tale, but shareholders don't know that, and boards don't know that, and a surprising number of the C-suites don't know that. Nobody has ever told them about stuff like transitive dependency exposure, because nobody gains when their heads explode as soon as they realise how precarious software is.
Hardware, though, hardware everyone can understand. That's the box in the server room. If we can't risk it going down, we get another box and we plug it into the first box. If we really, really don't want it going down, we build another server room somewhere (also with two boxes) and we plug that into the first server room, and then we have 100% uptime.
Except now we pay Amazon to run a lot of server rooms for us, so we can't possibly go down.
When Amazon does go down (and it's not due to our software, about which we do not speak) then somebody must be to blame. Heads must be found for the rolling.
The list of potential victims for the guillotine is not long. You can't really blame Amazon, because that seems weak. Unless you work at one of a very small handful of exceptions, AWS has a better technical reputation than your company. Also, at some point everyone up the chain signed off on moving more and more stuff onto the cloud in principle, and probably onto AWS specifically. Blaming Amazon would just make everyone look bad.
It turns out US-EAST-1 is still that linchpin, that critical node, that you and I (and not nearly enough of the C-suites) lie awake worrying about every night.
But nobody above you signed off on your choice of US-EAST-1 specifically, did they? And it would be fair to say that, by now, you should know better. US-EAST-1 famously went down in 2017, twice in 2021, and again in 2023. It made headlines. How in the name of all that is holy could you possibly have thought US-EAST-1 is a good idea? This is all your fault; security will escort you off the premises.
That looked like a plausible scenario for the couple of hours when we thought it was just US-EAST-1 this time, because surely, surely AWS had learnt its lesson and made sure that US-EAST-1 was not a single point of failure across identity services supposedly located in other regions.
But no. It turns out US-EAST-1 is still that linchpin, that critical node, that you and I (and not nearly enough of the C-suites) lie awake worrying about every night. Thus, many big-name services that do not pay to be on US-EAST-1 also went down, and there was nobody to blame.
This time.
US-EAST-1 dates from a time when you could just write “the cloud” surrounded by a cloudy squiggle on the whiteboard and everyone would nod along. We've lost that innocence in stages. The vendor started to matter for reasons beyond cost, then the capabilities envelope started to matter.
By contrast, strategic interest in exactly where your bit of the cloud is actually located in space, that arrived with a bang a couple of months ago. Suddenly, geopolitical risk needed to be quantified and projected, and now lots of people feel qualified to talk about location risk – and to ask pointed questions when things go wrong.
Even when things go wrong because AWS breaks some DNS entries and it is absolutely not even a little your fault.
Next time it might be only your region that goes down. And it won't matter whether that is because a bunch of undersea cables are all anchor-dragged at the same time, or because Google is running a big upgrade across its server management system, or because somebody breaks the bloody DNS again.
It will be because you chose it.
Don't say I didn't warn you.
Share