did you use a character controller on your XROrigin? That gave me a whole slew of bugs I had to fix. Basically any colliders that are not triggers now suddenly have the ability to push me around, including the things I'm holding. I had to autoswitch things to "is trigger = true" for all the colliders I didn't want to be pushed around by.
Not sure if that's it, but maybe related?