I'm guessing you were on room-scale VR, and we only tested with stationary. The player collider kinda works stationary, but it doesn't track the headset, so it gets messed up easily (we plan on patching that after the Jam is rated).
What to do is harder, and it would have been good to make them more clear. There are only two puzzles - one with potions, one with crystals.