Your game has an amazing concept! you should really make it complete and publish it.
Also gesture recognition worked really well for me! I think you can re-use it in many VR games, e.g. wielding magic spells based on gestures :)
Thanks so much! That's pretty much my plan, so I appreciate the encouragement :)