I'm sorry, I don't know what choices you made but Baxter is always meant to say quite directly that he's only a suitor for the season when he asks you out and it's essentially a fling. He shouldn't ever ask the MC out as though it's a normal relationship. If you missed the conversation about that perhaps there was a bug where it got skipped.
But I'm glad you enjoyed his story!