Blink is sometimes not working on my Weapon Set 2 (Staff)

Blink is sometimes not working on my Weapon Set 2 (Staff). It's working when you remove and insert the gem in the skills UI. But then after swapping to my main weapon to fight monsters, when I get back to my second weapon it doesn't work anymore and need to remove and insert the gem again