I really like the concept, and physics based games thrive in VR! Felt like a child 😃
Some minor feedback that would make it less finicky. Put the objects on another layer and disable collision between the player to not fly up.
When you grab object, especially the bigger ones, changing the attach point to the hand makes the placing easier. Occasionally when I grabbed the object, it snapped and pushed other things around.
But this is just minor, very good jam entry!