You have to talk to him about it the same time you find him beaten by the bandits after their raid on Bareshade. You only have one shot to encourage him to go.
Day 20 is when it happens as part of the story line after you cure the shopkeeper of his sickness (Which happens after you visit him the day after giving him the Spectral Sirup)
