The first thing I worked on was buoyancy, which is modelled for all the spars, so that's what you're seeing when you capsize. I guessed the distribution of weight around the ship.
Square rig physics doesn't seem that bad - perhaps it depends which corners you are willing to cut. I am modelling each sail separately as a wing (really a position, an area and a normal). I don't worry about, e.g., aspect ratio, I just have general lift and drag coefficients, tweaked until it seems reasonable. The only effect of sails on each other I model is screening each other from the wind; for this, for each sail I cast a dozen or so rays in the direction of the wind each frame, and see how many of them hit something. It's using whatever physics engine is built into unity4.