Game freeze everytime I enter originator's memory map

Details on problems:
Game will freeze after I enter the league content map portal, appoximately about 2seconds after entering the map. Everything is still running, animations also works fine. I can open up menu, inventory, or even passive tree. But I absolutely can't do thing, I can't move, attack, or interact with anything. the only options left are either 'exit to log-in screen', or 'exit path of exile', clicking on 'go back to character selection' option is also not working. This has happened almost 100% on these originator memory maps.
While having the performance window turned on, I can see that the 'network' latency keep going up infinitely, same as the 'lantency'. I have tried switching to other servers (japan, singapole, and california), it's the same, game frozen after entering the map for 2 seconds.
This has almost make the game unplayable for me, I am very excited about this league, but yet I can't do any league content. It's so frustrating, please help me !
Computer specs:
CPU: AMD Ryzen 9 7950X 16-Core Processor
GPU: NVIDIA GeForce RTX 4090
RAM: Corsair Vengeance 64GB (2x32GB)/5200MHz/DDR5
Game Settings:
Renderer: DirectX12, I tried vulkan as well, didn't help
Triple Buffering: disabled(default)
Multithreading: enabled(default)
Here's a picture with performance window on Imgur

Same issue, but I'm on Alleyways, the game freezes almost as soon as I enter and the I get "Waiting for game" and eventually I lose a portal...