It sounds like you picked up the "Commando" perk, either by taking Exhibitionist, or choosing not to wear underwear for several days in a row. As far as I know, there's no way currently to remove this perk without editing game files.
Based on player feedback, the dev mentioned he might revisit how this perk functions in the future.