One recommendation: I don't know how are you implementing this but I wouldn't rely on the engine's physics for this game. That would solve a lot of issues!
Yeah it's just the orbs (Area2D). All other collisions use custom logic.
I meant I woudln't rely on collisions altogether