I think the controller direction in VR mode is relative to the head direction since there is no motion tracking for an xbox controller or keyboard. It's definitely more natural to use the Oculus controllers if you have them.
