Same Skill, Different Hotkey But Different Weapon Sets Disables Gem Permanently

I want to use a skill, The same skill on left click as right click but have it do something different on right click associated with 2nd weapon set and passives allocated towards it..

If I Assign Left click to use weapon set ONLY 1, & Right click to weapon ONLY set 2..

this will unassign the skill from both Left and right click my, hotbar and will no longer be able to produce a thumbnail from this specific gem

I had to cut a new gem of the same skill re assign the actions and then switch back to my original gem

am i doing something wrong and this was some sort of intended function?