Unfortunately, that ship has sailed (pun intended), but you'll always be able to choose whether you want him to be a part of the "fun". A lot of my player base holds the Fisherman in high regard, so I'm afraid you're in the minority for not wanting him to join the crew.