It's really interesting how those words hit us, isn't it? I have almost the exact opposite reaction--for me, "panties" has somehow acquired an undertone of something a child would wear, which always makes me a little bit uncomfortable, whereas "knickers" is hot. Not saying you're wrong, just that it's strange how people pick up different senses with words.
Meanwhile in East-Asia, "panty" refers to lower-body underwear for both men and women. For men's boxer shorts, they're referred to as briefs, but the majority of Asian men wear the triangular "panty" variety thus heavily contributing to making the word "panty" a common unisex term. Panty just became the word for "underwear," and was heavily influenced by Japan during its colonial rule before ww2.
Like in Korea for example, the word for underwear is literally translated to "inside clothing" and can refer to both upper body and lower body, while the lower body pant-version is simply referred to as "panty" and is commonly used by both sexes.
Of course, the North-American perspective prefers to separate the underwear terms into masculine and feminine groups with barely any middle ground. Boxers/Briefs versus Panties/Knickers.