Can't remove jewel from passive tree

Please help! I clicked it using a mouse, and also xbox controller... it won't work and is stuck.

If it is a unique jewel that allows you to allocate other unconnected nodes (in a ring), make sure that those nodes are de-allocated before trying to remove the jewel.

Otherwise, just to be sure: did you right-click the socketed jewel with your mouse when attempting to remove it?