Buffs gems go offline the moment I change area!

Hello guys,

I am having this issue with the Buffs gems being deactivated the moment I enter a new area being maps or anything else.
The stats requirements are right, the points allocations are right, this started to occur the moment I chose to use Passive points in Weapon sets.
It works fine if I remove the gem and place it again.
But is annoying to have to do that every time I enter a map.
Is there any fix on this?
I have to say that both weapons meet the requirements, so is not that, and it works fine even when I swap weapons, but the moment I change area, the buffs are gone.
Mention: The buffs icons are on my bar, but the buff itself is not active, I don't see the aura around my character, nor the debuffs icons on the enemy, unless I remove and place back the gem.
So if you please help me fix this issue, I'd appreciate it!

Thanks and have a great day!

Later Edit:
I realized the issue also appears when I level up. So basically, any change on the character, disables the buff gems, and I can only reactivate them, by removing and placing the gems again!
The buffs appear back if I do that.

Hope this helps to fix the bug!

Thanks