Since only modifier seems to be the virginity, can't see it being enough to prevent after a point unfortunately. I have noticed that once I take virginity despite them being unwilling, they accept it, so it seems to me that there may be an issue with how much being a virgin affects their consent for such actions, since without virginity they're fine with it.
May be totally off, since I've no idea how the game's code works, but based on the only way to get consent being taking their virginity in first place, seems the most likely issue.