i appreciate that there are options for other partners/ntr in red and think the options for the other crewmembers are fair since they are not fully creepy. However is there a possibility that you can make it so you can choose whether the fisherman joins your boat because I just really don't like him but I can understand if that might be too much work.

Unfortunately, that ship has sailed (pun intended), but you'll always be able to choose whether you want him to be a part of the "fun". A lot of my player base holds the Fisherman in high regard, so I'm afraid you're in the minority for not wanting him to join the crew.