I've sometimes noticed that while I have like 2-3k in the bank and dismissed members, some guys will go missing though.
Followers waiting around at points of interest have a small daily chance to leave to go find other work. Your party's money has nothing to do with that.
