Yes it is a typical problem for throwing in VR. What I'm saying is you need to do something like aim assist in video game consoles. Just enough to make the player feel like they are really good, but not so much that they notice you are cheating for them.