Make sure to keep the bandit's camp's morale high, it should be >25. Barrel visiting our house is just so random and doesn't happen very often. But, once you get 15 like (iirc) with him, you can find him in the tavern & have sex with him.
