I think he gives you a chance further down the plot to go back and serve him I think. During the Orc War event. I think that's what it implies, I wouldn't know since I've served him from day 1 of meeting him.
