I only romance women, so I tend to steer clear from NBs unless I know their gender.
Honestly that response to me indicates that you are interested in women. why are you trying to force gender on a character without gender? I'm a cis queer woman so maybe my viewpoint could do with input from people of other genders and representations, but I do believe being non binary means you don't feel you are either or any gender. So pursuing the gender of a non gendered character doesn't make sense. It just feels like you're trying to force a gender on them.
I think perhaps, this what other people might be inferring and that is why they do not like your comment. Im not psychic though, I'm not always the best at understanding some topics but I am trying. I don't mean to be insulting or argumentative either. I seek an understanding of motivations, social knowledge and academic knowledge