Ah, somehow I never saw this comment. I disabled physics update and then manually run it myself using Physics2D.Simulate with a time step determined by an easing function. In this case the function is CircInOut.
You can come and pick my brain in our discord server if you like.