Whispers of Doom - Weapon Set Allocation (Double Curse)

It seems that Skill Points can not be applied to Whispers of Doom when choosing either Weapon Set 1, 2 or Shapeshift.
Trying to apply it for any other passive Skill allows me to apply it.
Just not for Whispers of Doom in particular.
Whenever I uncheck the Weapon Set Allocation, i can freely choose Whispers of Doom again.

I also dropped Blasphemy out of my Skill gems and see if it interfers with it.
It does not.

Weapon Set 1:



Shapeshift Weapon Set:



Applying it regularely:

del