Random people joining our party

My wife and I were playing this morning, standing in town and had some random person named Udder cheese join our party. It would not let us remove them without exiting and reloading the game.

My question is, why can random people even join a party uninvited, let alone not being able to remove them?