You don't have to turn them off but consider (if you still have the layers for it) having the player controller on its own collider layer and having a collision layer that doesn't collide with that layer (you can set this up in the Unity physics matrix). When you pick up an object shift it over to that player-collideless layer, then shift it back when you set it down. That'll keep the physics but you won't end up with issues.