It's likely due in part to how I have my VR space set up personally. I have my chair and desk facing the opposite direction of the direction of the VR space. I did kind of like wandering around inside the town.
