You can dismiss them by pressing the X next to them in the building.If adding them after that doesn't work, it's a bug.
no i mean add them to your party not dismiss them
