It may be a bug or it may be that you interacted with the slavers which makes Robin leave you (he doesn't like slavery for some reason... weird)