Hey, thats a really nice concept !
I once made a game that works in a very similar way (in 2D) and encountered some interesting problems and solutions. I encountered one of theese problems while playing your game. A couple of times the previous me would not take the exact same path I took and I had to restart.
Im guessing you are applying the same veloctiy values to the copy that were recorded ? I think thats a cause of inaccuracy which sometimes let the copy run into walls and take wrong pathes.
Anyways the game was playable and a fun concept that I think works particularly well in VR.
Great job !