No, you just need the right amount of combat prowess I think that "skilled" is enough to start dealing damage to them
It influences some scenes like, when the bandits attack Bareshade and you enter Bernard's shop when he suprise attacks you, your character will react instinctually and do an over shoulder toss, or while you're having the training session with Roushk if you're skilled enough you'll need less strength stat to win, not to mention that your dealt damage increases as your proficiency does, same with teasing