You can dismiss them by pressing the X next to them in the building.If adding them after that doesn't work, it's a bug.
no i mean add them to your party not dismiss them
They can't still be a guard if you want them back in the party. If you've dismissed them from being a guard, on the overview page, there should be a "edit party" button at the bottom.
