IIRC, meet him at night and he basically offers it to you (something along the lines of him saying "Hey, join me in the hot spring if you want"), and the option to join him in the hot springs becomes available.
Thank you so much. Maybe it's a chance thing to get it since I sometimes meet him at night in his hot spring. What I also remember is getting it in the day, and I probably got excited, so I immediately returned home to skip time and meet him at night.
