I know it's hard with physics. Have you tried setting the dragged body to kinematic mode temporary? Just an idea from the top of my head
Yep I did try that, but I'm a bit confused about how that's supposed to work in code because it's not like changing the body mode all of the sudden gives you access to the normal kinematic body methods like move_and_slide... There's definitely a better solution then what I landed on, who knows if I'll ever get to implementing it though haha.