Game Performance - are < 20 fps drops considered fine?

I recently upgraded my PC - the old i5 3570K was getting a quite long beard - and naively thought that PoE would now run fine on a 9800X3D with an 9070 and 96 GB 6000-30-36-36 RAM. It may not be the best hardware currently available but is by no means slow. The performance graph also shows that the cpu is the bottleneck when the fps go down.
Some Mechanics still make the game a slideshow.
Rituals somehow are bad even with a seemingly not so high monster count.
Blight was always bad i think and due to extreme possible mob counts might even be acceptable somewhat. But please let us click the tower buttons on low fps too. I keep running towards the tower - into the mobs usually - because the click went 'through' the tower build/upgrade button. Happens quite a lot when FPS drop below approximately 15 but starts already at about 20.

Shrines are mostly fine despite instantly spawning large groups which seems to be a bit of an exception.

The worst is still rituals since 'normal' players cant just idle tank everything while the screen basically freezes and fps approach something like 5 while the map boss spawns besides some rarey with nasty things like lightning totems, exploding crystals, you (should) get the idea.

If you think the game just shouldn't be run on a potato, then please tell what would be good enough. What do you test the game performance on?

On a personal note, i started PoE in 2013 i think. My first character was a 2h Mace marauder with heavy strike and sweep, which was really fun back then. After the game got increasingly faster and the mob count went up i switched to an arc witch in order to utilize the auto aim of the skill since even leap slam and sweep was getting annoying with spread out, fast sometimes melee avoiding monsters.
The game continued going the 'more, harder, faster' road and now i consider minion builds the only viable ones for me. The only thing that can keep up with ai mobs is either overpowered character builds or my ai mobs. Especially when the game suffers from connection latency, server lags or local slideshow.

Now when playing with a friend the most threatening thing is poor visibility due to all the mobs and effects since it's almost impossible to make out ground effects, exploding crystals etc. Not finding the Map boss on the screen is a normal thing too. So the only safe way of playing is to keep moving, stay out of range (may also mean out of sight) and let the minions do the work since they are not impeded by low fps, splashy visual effects and reaction times.

I sometimes miss the game from 2013/2014 where it was not overloaded with visual effects, not overcrowded with hasted mobs where a marauder could still be just 'not ready yet'

The game definitely has performance issues, and many times a lot of it is server side and nothing you can do.

But with your hardware, you shouldn't be having as bad of a time as you are having. I'm running a RTX 4060 on a 5 year old PC with 16GB of ram and have no issues with ritual.

I would suggest:

1. Verify gateway latency on login / try connecting to a different server
2. Make sure your drivers are up to date
3. Try messing with some settings. Theres lots of YT vids on this. Heres a semi recent one to try.

https://www.youtube.com/watch?v=9M_lKznMn38

You can also go into the game config and disable all sounds for a huge performance boost, but its a very weird way to play IMO. I don't do it, but there are plenty of videos on it out there.