weapon swap breaking spirit with minions

so i start with say 13 minions, i swap weapons to despawn them and swap back, when this happens i now i only have 10 minions and only enough spirit to put 2 back. setting all minions to 0 in this state does not produce a full bar of spirit, theres just a chunk missing, unequipping my gear and re-equipping it gives it back ~70% of the time, but thats not viable mid combat, nor is having to re-spec the missing minions even if all the spirit was readily available. then once i finally get them all back, the party menu shows double the amount actually available (doesn't erase the originals from memory im guessing? as they're permanently there until i portal, just the portraits and number, no actual minions being duped here).
anyone else having this issue? the math on spirit lost isnt even adding up to missing the weapon spirit %/skills on the weapon so i'm really not entirely sure what the real issue is. it's incredibly annoying though.

edit: video example can be recorded if this isn't worded well, there's a lot of little broken parts here lol, its reproduceable every single time