Could you allow the ghost car to continue moving while in spectate mode, but then to reset to where it was when you entered spectator mode when you leave spectator mode.
When watching another ghost from the leaderboard it is possible to pause/unpause when using the spectate mode by pressing space.
It's not possible while playing though.
Oh I'm playing on my school computer and the leaderboards are completely broken because I'm using a proxy.