Minion Weapon Swap Bug

I have a bug that is causing my minions to only appear every other time I swap to my scepter using the weapon swap. I don't know if it applies to other types of minions since I am only using skeletal warriors via said scepter.

*Edit*

To clarify further, it only happens when there are enemies. It doesn't happen if I switch any nothing is hostile around me.

Do you play on mouse+keyboard or controller setup? I play on controller and this bug happens to me but once I switch to mkb it stops happening, seems to be a controller bug.