Hey!
Sorry for the delayed reply. I've been a little busy lately.
I'm using the VRTK framework so even though I don't have an Oculus Rift I thought it would work anyway. In the coming weeks I'll see if I can get it running on the Oculus quest and maybe that will improve your experience as well.
Can you tell me which controllers you see in game? Is it the Vive controllers?
And yes, please, if you have any ideas I'd be willing to listen.