Performance Improvements

Path of Exile receives significant content and feature additions several times per year as we roll out new expansions and challenge leagues. Keeping the game performing well with these additions is an ongoing challenge. This manifesto post details some of the issues and what we're doing about them.

Server Performance and Lockstep Stuttering
Lockstep mode works by only stepping the game simulation forward when complete and correct data has been received from the server. This is why it doesn't go out of sync - because it's making exactly the same decisions that the server did. Sometimes, though, the necessary information hasn't been received in time and the current frame has to be rendered without stepping the simulation forward. This small stutter is different than a graphical client freeze because particle effects continue, but everything stays locked in place. Once the correct data has been received, the client accelerates time slightly to catch up with the server.

There are two reasons why data may be late - internet latency and server performance.

Some users have volatile internet connections. Fluctuations in their latency manifest as lockstep stuttering. When we become aware of network issues with ISPs, we report these problems to those ISPs to get them resolved so that people can play properly.

Performance problems with our game instance servers can also cause lockstep stuttering. While the runtime performance of the server software mostly affects the size of our server hosting bills, complex combat situations can occasionally take too long to calculate a frame of results and cause that frame to stutter.

We recently identified some combat-related performance issues that we are targeting for fixes. We don't yet have a solution for these, but we'll deploy it as soon as we do (in the coming weeks?)

Memory Usage, Synchronous-Load Stuttering and Load Times
I'm going to start by explaining how Path of Exile works in terms of memory management. This will make it easier to understand how we're improving it.

In order to draw an art asset on screen or play a sound, it has to be loaded into memory. Path of Exile has around 7.7gb of assets, and we can only load a subset of them at once. Loading an asset causes the game client to pause while it's being read off the disk. This is what loading screens are for, and there are two types of loading screen that we can load assets at.

The loading screen that comes up when you initially run the game is most appropriate for assets that are very likely to be used in the game session. This includes things like the core character classes or items.

Loading screens that appear before levels are best for assets that occur during that level, like environment tiles, monsters and the skills they use.

If an asset isn't loaded on one of these screens correctly, then it stalls the game client to load when it appears for the first time. For example, opening a Strongbox can generate a random Rogue Exile. If the effects or microtransactions that that Rogue Exile uses aren't already loaded, then the client stalls, potentially in a very dangerous way.

Preloading assets has a significant cost: Everything that is loaded at startup increases how long Path of Exile takes to load. It also increases memory usage. On 32 bit versions of Windows, processes are allowed to have 3gb of data loaded. On 64 bit versions of Windows, Path of Exile can load 4gb of data at once. If this is exceeded, the game crashes because it can't allocate more memory. It can even crash before this is exceeded, because of memory fragmentation issues (if there aren't holes large enough to fit the files in, due to a long game session).

On the other hand, if we opt to not preload assets, then there's significant stuttering that occurs during gameplay when assets are encountered for the first time. On slower hard drives, or when virus checkers or disk fragmentation are in play, the freezes related to synchronous loading can be quite long. This is unacceptable during combat, for obvious reasons.

For many years we've been maintaining a carefully curated list of assets to preload. We were trying to strike a perfect balance between loading the things that are needed, but not loading too much that we'd have slow load times or exceed our memory allowance. As our data size rose, we've had to develop more and more complex systems for managing which assets should be in memory at any time.

We've recently been doing a thorough sweep of all of our assets to work out exactly when the best time to load each is. The first part of these changes were deployed in 2.2.1e, and more are coming in 2.2.2 next week and subsequent patches. The changes so far have reduced both synchronous-load stuttering and memory usage/load times. The rare crashes that occurred changing areas on 64-bit systems are now gone.

At the same time, one of our performance-oriented programmers has been working on a new asset management system that can load assets asynchronously. This will result in an additional reduction of load times and further reductions to stuttering. This system is not yet ready for testing, but its deployment will hopefully be later this year.

Audio Overhead
Some players choose to disable audio to improve client performance and reduce memory usage.

We're currently working on switching to the FMOD audio system, which is an excellent commercial audio engine. This will both offer client performance improvements, but also hopefully reduce memory usage by allowing us to keep audio files in a compressed state in memory. We expect to release this improvement alongside the 2.3.0 challenge league.

Draw Calls
Every object that has to be rendered in a scene is a Draw Call. Processing all of these consumes the CPU (in addition to whatever rendering later happens on the GPU). On Windows XP, this is especially slow. Reducing the number of Draw Calls helps improve performance in situations where the CPU is the bottleneck rather than the GPU.

We recently found a problem that affects certain types of lights in Path of Exile. It would dramatically increase the number of Draw Calls in the scene when those lights were present. We have already deployed a fix to this (which helped reduce Draw Calls) and have a Labyrinth-specific fix being tested for an upcoming patch.

64-bit DirectX 11 Client
In the future we plan to release an alternate Path of Exile client that is compiled as a 64-bit executable and will use DirectX 11 rather than DirectX9. This client will require Windows Vista or above (and a 64-bit CPU). The old client will still work for Windows XP or 32 bit machines, but won't have the performance improvements that come with DirectX 11. This release is planned for 2017.

Realm Stability
We have had an unacceptable number of realm stability issues in the last few weeks. Our network architecture involves a central set of Master servers in Dallas, TX. Our gateways around the world (in Europe, America, Brazil, Australia, etc.) connect to these Master servers. If there is a networking incident on the internet somewhere that disrupts this connection, then users on the affected gateways are disconnected. If the networking incident occurs in or near the data center that our Master servers are located in, then all users are affected. Recently we've seen disruptions like these every few days

These networking disruptions are generally outside of our control, but our end responsibility is for us to deliver a good gaming experience for you. We're aggressively looking into what options are available to us to solve this. It may involve migrating the Master servers away from their current location, for example.

The good news is that aside from those networking issues, stability of the actual servers themselves (as opposed to their internet connectivity) is quite solid at the moment. We've had issues in the past where parts of our code struggled with the recently increased player load, but those problems are all resolved now.

Conclusion
Normally we'd make these improvements without drawing attention to exactly how the game internals work or what was wrong. There has been a lot of discussion in the community about performance recently, so we hope that this post helps explain some of the technical challenges we're solving and some of the promising improvements that we hope to deploy. As you can see, some of the improvements will be very soon, and others will be released over the coming months. Several of our programmers are now permanently assigned to performance work, and we're already reaping the rewards of this investment. We greatly look forward to continuing to improve Path of Exile's performance as we release some of the exciting content and challenge leagues we have coming up.