I had this issue, it does not grey out until prison but the mistake is earlier in quest. there is a hidden "meter" of affection and if you do not have enough with a character at certain story points it will fail. The prison is the fail point not the mistake point
