With walking Calamity active and lots of mobs on screen, CPU can lock up

With walking Calamity active and near abyss or ritual or any large pack of mobs my CPU will eventually hit 500ms and my ping will hit 500ms and continue going up. The game runs at 2 fps and does advance every few seconds. If I exit to character select I can play again but any activated mobs like ritual will be gone. Occasionally I can escape the abyss area when it starts getting to 2-300 ms and it will settle down but it gets worse again when I get close.

Using a Ryzen 5700x3d and amd 9070 on bazzite 6.17.7(linux) with 16GB of RAM.
I use FSR3 Quality with no dynamic resolution, windowed fullscreen at 1440p.