I think there is a bug with the framework I used (VR Interaction Framework). I noticed it too - sometimes when you play the game, it doesn't appear to raycast correcly. :/
