At end of day, for each slave in a guild there is a roll with a 20% chance to remove slaves from the top, so the more slaves that are in a guild the faster they are removed. Though in general it's best to pick one slave guild to sell all your slaves and buy from the the others.
If a guild has less than 4 slaves, then it gains one slave. If it has less than 6 slaves, there is a 75% chance to gain another slave.
Ending the day more, will result in more changes. Otherwise, buy all but 1 of the slaves in a guild(Umbra is all but 3), then save and reload, it will be restocked with 4-6 slaves.