I don't remember any of this actually happening in the game, so you either mixed up the story of another game or you interpreted things in a different way than I meant.
The only thing I can think of is the pool scene, where the girls take off his swimming trunks as a joke, but it has nothing to do with being better at sports. In any case, if our protagonist had been in trouble, the girls would have confessed that it was their fault. They would never wish anything bad for the protagonist