Map portals randomly dissapear

It happened to me three times so far. Map portals randomly disappear. I just portaled to my hideout to refill my flasks and now I can't go back.

I have no idea if any specific actions trigger it or not and unfortunately can't provide much more info :/