Last Friday early morning European time, our US-based brethren informed us that many players who tried to access CoN found the ominous phrase Bad Gateway painted across their screens.
(For those of you who want the concentrated information, skip to the official TLDR snippet at the bottom of this post).

I’ll spare you the details around the nature of this bad omen. The introductory word ‘Bad’ in Bad Gateway chucks subtlety out the window, and adopts self-explanatory rhetoric. Coincidentally, this gate is the not-so-distant relative of Gatekeeping. Another phenomenon that generally warrants the same reaction.

“Get on with it” I hear you say.

As the team scrambled to find the cause, it was clear someone left the oven on and eggs over-beat.
You see, Bad Gateway is a very generic term which encapsulates server-proxy issues. Basically, you went to a restaurant that never got it’s delivery that day. When the victims on-call staff realised that the oven was on, but empty, then it became apparent that we needed to retrace our steps.

And thats when we found every culinary chef’s worst kafkaesque nightmare. The eggs were beat to a pulp fiction the likes even modern theatres would not show.

(In essence, for those not enjoying my kitchen analogy; a group of servers hosted by partners went belly up. (Not from overeating))

“Why”? I hear you ask. Continuing with the kitchen murder-mystery analogy;

Evidence clear of eggs and whisk murderously tossed, the kitchen chefs had a clear warning drawn out in egg: Shipment that week missed the CoN diner, for reasons outside of our control. With this knowledge in mind, the team set out to resolve the shipment issue by other means, and are hiring a designated watchdog (I’m not joking, it’s called datadog ) to catch these situations as fast as humanly possible.

TLDR:
Last Thursday issues cropped up from a group of running webservers provided by a third party, which caused an overload. Bytro & Dorado worked in the early hours of the morning to resolve it. This isn’t something that is in our control sadly, but what we are going to do is have a lot more monitoring tools in place to catch these instances, much sooner. We’re also investigating other instances where and why this would happen.