I'd say it does matter. Relatively small, but it does. For example sex with Harold in the inn looks differently for dominant character and for any other, women can't dominate some characters at all, for example Harold for that matter. Bottoms have exclussive content with characters like Dulrig or Hayden, some scenes with characters like Feliar, Roushk or Jester are different depending on your dominance, some characters like Tank or Barrel react differently for such characters. There are some scenes with the latter when you can be a bottom yet dominant. Enemies like Nymphs also will react differently to women or gay characters than to women-liking men.
As I understand your point I just want to ask you, how do you think it would look like if you chose heterosexual male as a character. Libido damage would almost never matter as most enemies in the game are males. Also, I can't imagine situation if you would lose and you would be r-worded by male enemy. In other games like this it is still understood that sex is pleasurable then as you are bi/pan-sexual even though. Here, if the character would be hetero, wouldn't this be, well, I don't know, distasteful? Just a little bit? Not to mention, most of game content is male-centered so you limiting yourself to sex with female characters would be really reductive.
I'm not saying that you're wrong, but some mechanics are a lesser evil here and kind of necessary. Also, don't be afraid to experiment with the game. There is also quite unique content with Logan if you're a virgin for that matter. So there's a lot of possibilities, don't let your preferences limit you from finding more about it ;)