Hey, I had the same issue and honestly it just comes down to not having enough heart points with him, you may wanna check the guide to see the best choices to make and restart or restart and just look at how each choice effects your affection rating with him before continuing! If you're going to do Akechi's moon route I suggest the same thing, I struggled hard before glancing at the guide.