Minion counter desync when switching weapon set

In weapon set 1, I have a sceptre that grants Level 3 Skeletal Warrior Minion. When I switch to weapon set 2, which does not have the sceptre, the count for the Level 3 Skeletal Warrior Minion does not disappear. When I switch back, the 3 skeleton warrior minions are respawned and added to the count. As such, by repeatedly switching back and forth, you can continue to further desync the counter.