Yeah, that scene is kind of iffy. I remember it being close to a dealbreaker.
The girls' complete lack of consideration for the consequences he might face is quite offputting and cost them a lot of sympathy. Whether they intentionally put him at risk or just didn't care to think about how their actions might affect him makes little difference, both are revolting.
Even if you ignore the possibility of being mistaken as sex offender, they forcefully exposed him in public against his will. That's sexual assault, along with a number of other offenses. This is made worse by the fact he would not get away with such behavior. I can't really see a way to justify that even if they were to "fess up" about their involvement.
You can't blame him for evaluating the scene based on the events that objectively happen. If anything, you misjudged the severity of that "joke", and did not manage to strike the tone you were trying to. That scene might be worth reevaluating.