If you did the lighthouse puzzle to the fortress, then it's possible the grave stone prevented you from seeing the X of glowworms clearly because you're too close to see the full X.
removing the grave stone so you can go far enough to see the X is part of the puzzle I think
