You can make a prisoner a servant by clicking a promote button in the collection tab of the items menu. You can see the button below their avatar after clicking on a prisoner. It's a chevron icon. It only works if they have a positive attitude towards you, though. So I think that means it only works on characters you've saved and brought with you to the summit, not necessarily prisoners, at first. You also need to remove them from your party if they're in it. You can click on their little portrait on the left of the screen and remove them from the party.
There seems to be a way to make prisoners like you and I suppose you can promote them to servants afterwards but that's not working for me yet.