Unable to weapon swap when playing "Disciple of Varashta" ascendancy (Djinns)

When you assign Djinns to a specific weapon set, and other generic skills to a different one, it's impossible to swap to the Djinn Weapon Set automatically by pressing the respective skills.

Even manually trying to swap causes a weird delay before the swap is complete.

The game probably forces Djinns cooldowns when you try to swap to their weapon set.

I tested it with the feature that allows the same weapon/off-hand to be applied to both weapon sets.