I had this issue, it does not grey out until prison but the mistake is earlier in quest. there is a hidden "meter" of affection and if you do not have enough with a character at certain story points it will fail. The prison is the fail point not the mistake point
I am not one to use walkthroughs until after, but maybe look up to where you are? See if you had an unsuspected fail much further back. If you go back... a year? I failed with Alex in .5 because I didn't say her pen looked great.
Pen-point accuracy VN's usually turn me away, but the story is just too good, OiaL as well