It's possible to lose all the unique characters based on quest choices (I think). Even if they are in your mansion they can disappear permanently when they are sold, delivered, executed, released, and so on. Zoe and Ayneris can be missed entirely. I'd suggest a restart and replay if that bothers you more than losing your current mansion.
For Ayda, Ankmairdor tracked the bug down https://discord.com/channels/482888320117112843/482888320117112847/7906611647612...