Yea thats a nice Idea.
I ended up with a hybrid version that would track both and use whatever made more sense.Not sure on how performant it is in VR tho.
