I've exhausted dialogue with literally everyone but the counter claims I have one more refugee to save, I did the derby by paying two debts and ensuring that the last one won, so I'm assuming this is why, even though he Won and should be free?

It's not one of the contestants then. Maybe the slave in the bottom right, that's the one that people miss a lot. Or the gambler under all the blackjack tables.