You can fight him once in Fishing village during Northcrest conspiracy event. After that you can fight him whenever he not in Bareshade at Riverbank. And beware about your first fight, it has a long-term effect to Gunnar relationship too.
