She isn't coming to your castle, she remains in the Spire.
The answer is in the walkthrough:
Q:
How do I solve the riddles in the dungeons during the „Fragment
Hunt“
quest?
A: In Lualombo, you need to move all switches to the left side to unlock the dungeon at Sinasoka, or all right to unlock the dungeon at El Arma Gravesite. It's impossible to have both unlocked at once. The door at Lualombo requires the combination 5571, which is the count of the switches in the dungeon at Sinasoka. The switches don't need to be used, you only need to count them to get the number.