You just talk to him until your like is high enough, then while talking to him at night you have an option to talk about the hole in the fence... from there it gives you the option to go with him there or to leave. (Or at least something around those lines)
