There a scene in Fishing village that you can really fight him. If you tease him until his libido reach 100% and choose to fuck him, he will hate you (get a black heart) and he will avoid to meet you (so scene with him will no longer trigger).
Have you visit Iron Blood tribe village yet? There should be at the top of the map. And another question, have you ever meet Groth while you explore wasteland?
About that, there should be a option 'look around'. You need to look around the area there untill you find him since it RNG. So there a chance that you wouldn't meet him in several try.
After unlocking the fishing village you can fight him once a day in the rivers bank. The majority of his sex scenes don't get triggered if you have a black heart, but the caravan scenes near the bar and the scenes when you prostitute yourself at Bareshade are some exceptions.