Yes it's a classic sailing model dilemma where one improvement causes problems in another area. Getting underway is already reaaalllllly slow from a gameplay perspective (I really have no idea what's realistic here). More importantly it would make tacking a lot harder if you increased low speed drag.
How long should a ship take to slow down? Well in a small boat, you coast to a stop very rapidly, whether power or sail. The ship stops rapidly when tacking, due to wind pressure on the sails. My gut tells me that stopping due to water drag along should not be more than 2-3 times slower.
All in all, it's the kind of situation that would make me want to create some sort of artificial braking effect when the sail area is under a certain threshold.