I used to max my appearance too cause wanted to try win all battles by sex in my previus time playing, but didn't manage it. Other stats are very useful, they let you use different styles in battle, using meele and ranged attacks, avoid being killed by one punch, cast more powerful magic too. so now i upgrade all of them (appearence is still my favorite).
Also a little tip: if you use sword in battle, you can speak with Feraldo, if you let him join the crew. He can teach you some useful skills