After unlocking the fishing village you can fight him once a day in the rivers bank. The majority of his sex scenes don't get triggered if you have a black heart, but the caravan scenes near the bar and the scenes when you prostitute yourself at Bareshade are some exceptions.