Hmm, it looks like there is a bug the only time you will be able to visit his house is 6 PM and 7 PM.
If you still can't access his house even during those 2 hours then the cause is unknown to me. Following through and having all the dialogues with him should lead you to his house naturally.
