Thank you very much for playing and for the feedback!
The only way to not have the ghosts wait around, is to make sure you don't wait around in your present-timeline as the ghosts are 1:1 copies. This seems to be a flaw in the design, so it is something we can take closer look at.