Ultimatum Launch: Server Issues and Streamer Priority

UPDATE: Server stability issue appears fixed. Be careful with your database page sizes, people.

Hey everyone,

It's been a long day but we wanted to put together a few thoughts while we have a moment waiting for our next server fix to build. This launch has been rough, to say the least. In this post, we plan to address both the ongoing technical realm stability issues and the conversation around streamers getting priority in the login queue. We are sorry that this is being addressed so late in the day - we have been giving the server issues absolute priority and haven't had time until now to write up this explanation.

Let's start with the technical issues.

Immediately upon launch of the league, we could see that the queue was running incredibly slowly. At the rate that it was emptying, it'd be at least two hours to get everyone into the game. The reason was that when players logged into their accounts, the server would migrate any previously un-migrated Ritual characters to Standard, which can take quite a lot of time to do on-demand (as much as three or four seconds per character in some cases). Users who had already logged in since Ritual ended were already migrated and were nice and fast. Normally, we run a "trickle migration" process in the background that performs this action on every account over the few days between the last league ending and the new one starting. Due to human error, this process was not run and hence the queue was unbearably slow to empty. (We have since codified this step into a QA checklist so that can't be trivially missed again in the future.)

We realised that a solution was to disable the Ritual-Standard migration entirely, which would result in the queue emptying very quickly but players would miss some Standard progress until we run it again later on. This solved the queue speed issue by around the one hour mark. At which point, the realm freaked out and dumped most of the players out, then continued to do this roughly every ten minutes or so for the rest of the day.

This wasn't good. At all. Aside from catastrophically ruining our launch day, it completely mystified us because we have been so careful with realm infrastructure changes. We thoroughly tested them internally, peer code reviewed them, alpha tested them, and ran large-scale load tests up to higher player capacities than we got on launch day. We even went so far as to deploy some of the database environment changes to the live realm a week early to get real user load on them just in case. But yet it still imploded hard on release.

I'll spare you the blow-by-blow of the hundred changes we have made over the last 12 hours, but we have been trying things one at a time in order of likelihood to fix the problem. There is one change we have been leaving for last (because it requires some downtime), but we have exhausted everything else we can think of, so we're trying that next. In the next 30-60 minutes after posting this, there will be roughly 30-60 minutes of hard downtime to make this change. We are optimistic that it stands a good chance of resolving the issue. (Note from the future: this did fix the issue!)

We will continue to work on this issue until the servers are working perfectly. We know the Path of Exile realm can handle this much load, it's just a matter of divining what subtle fuckery is causing the problem today.

Some players have also become concerned that when server issues occur, items are occasionally duplicated or destroyed when placed in a guild stash. This is a longstanding consequence of how our guild stashes work and generally isn't of much concern because players can't induce server problems and can't control whether the item is duplicated or destroyed. We are keeping a close eye on this of course.

So while this was all going on, we managed to also commit a pretty big faux pas and enrage the entire community by allowing streamers to bypass that really slow queue we mentioned. The backstory is that we have recently been doing some proper paid influencer marketing, and that involves arranging for big streamers to showcase Path of Exile to their audiences, for money (they have #ad in their titles). We had arranged to pay for two hours of streaming, and we ran right into a login queue that would take two hours to clear. This was about as close as you could get to literally setting a big pile of money on fire. So we made the hasty decision to allow those streamers to bypass the queue. Most streamers did not ask for this, and should not be held to blame for what happened. We also allowed some other streamers who weren't involved in the campaign to skip the queue too so that they weren't on the back foot.

The decision to allow any streamers to bypass the queue was clearly a mistake. Instead of offering viewers something to watch while they waited, it offended all of our players who were eager to get into the game and weren't able to, while instead having to watch others enjoy that freedom. It's completely understandable that many players were unhappy about this. We tell people that Path of Exile league starts are a fair playing field for everyone, and we need to actually make sure that is the reality.

We will not allow streamers to bypass the login queue in the future. We will instead make sure the queue works much better so that it's a fast process for everyone and is always a fair playing field. We will also plan future marketing campaigns with contingencies in mind to better handle this kind of situation in the future.

It's completely understandable that many players are unhappy with how today has gone on several fronts. This post has no intention of trying to convince you to be happy with these outcomes. We simply want to provide you some insight about what happened, why it happened and what we're doing about it in the future. We're very unhappy with it too.

UPDATE: Server stability issue appears fixed. Be careful with your database page sizes, people.