You have to speak to him during the day after walking up to him naked.
That will trigger a scene and a fight and then he'll offer to let you into his home at night.
