That one choice doesn't "ruin" the entire relationship. Your previous choices with X were likely incorrect as well.
Also, getting an "unsteady relationship" in CH9 doesn't mean that your entire relationship is ruined. There's literally 15+ more chapters to be added in the story - this is just a first indication of how X feels towards your Crown.
You can course-correct in future chapters. Try thinking more about the kind of character X is and which choices they would appreciate. The romance routes depend entirely on if you are able to understand your LI.
[Spoilers for CH9]
Edit: To clarify, X always leaves at the end of CH9 no matter what, even if your relationship isn't unsteady. It's part of the story.
Your Crown and X have literally known each other for a week, and X is a mercenary who needs to work. There is no "break-up", your Crown is not in a committed relationship with X yet. X is not going to stick around the palace just to romance your Crown lol, they have their own stuff going on. You'll see them again in future chapters