The physics in this game are real wonky, and I don't envy the situation you're in with them as I've had endless trouble getting Unity's physic systems to work how I want them to. Whether using 3D (Havok) or 2D (Box2D) in Unity, getting them to work right is far more art-form then most programmers like to admit. There's a really great lecture on the finer aspects of managing either physics engine in Unity at https://www.gdcvault.com/play/1021921/Designing-with-Physics-Bend-the.
I've had limited success in 3D with forcing manual movement calculations into a Rigidbody.Moveposition() in a FixedUpdate() while having Rigidbody components with fricton-less physical materials and in worst case scenarios making their velocity Vector.zero every FixedUpdate, but that's not a great fix. XD
Edit: A sidenote as I had an issue with it, make sure that if you're using physics to move your player character that you also have any fancy-smoothed camera movement also happen in FixedUpdate() to prevent the character jittering. Update and LateUpdate won't sync quite right. Sorry if I'm being redundant or unhelpful. xD