Ah, I see. You missed his offer to be invited to his house.
He should offer you to come visit again in the evenings. Maybe after the "mine quest" a bit later!
I'm afraid you might have missed the opportunity then. He should offer you whenever it's sunset (the sky is orange). There's two opportunities, once after the slug den quest, and once after the mine quest.
If you have any old saves to go back and witness these scenes, feel free to. But ultimately the only thing you missed out on are some relationship points and character backstory / exposition. No lewd scenes or anything.