Reproduction is a foundational element of life. It is inconceivable that any sexed species, nevermind an intelligent sexed species, could evolve without developing an awareness and concept of a biological phenomenon that is integral to sexual reproduction: that individuals come in two types with differing roles, one of each being necessary. This need not imply anything else: things like grammatical gender and gendered behavioral norms need not necessarily follow from sexual differentiation. But they would surely be aware of such differentiation and have a vocabulary to describe it.
It feels like the author decided they wanted a non-binary species for political reasons without considering what that actually means. They could have created a species where biological sex simply does not matter beyond the rudiments of sexual reproduction. Members of such a species would not self-identify as non-binary: they would self-identify as male or female and barely give the matter further thought. Self-identifying as non-binary is a conscious denial that the binary applies to you, which would be insane if referencing biology; it only makes sense as a rejection of something else, namely, norms associated with a person's sex. An entire species would self-identify as non-binary only if they first developed gendered norms, then chose as an entire species to deny those norms, and were still living in the shadow of that denial. I should like an explanation of how such an event came to pass, although I doubt anyone could create one that I'd find plausible.
Of course, I doubt that was really the author's intent; they probably just wanted a species without gender norms. It is ironic that they chose to do so by making it "non-binary," as that very concept implies said norms and therefore perpetuates the species' subjugation to what the author was trying to eliminate.