Haha, oh yes. I remember with dread long ago trying to make a lantern hanging in a chain with rigid bodies and constraints. Things went flying all around, it was madness.
I guess for this case maybe they could just be disabled on toggling mode? Not sure. But game physics can be very frustrating!