I'm not 100% sure, but I think that in the "Hunting Contest" Quest, you actually can raise his affection to the MC.You just need enough time left for the option to show up.Not sure if the option is bound to some past choices though.
