Indeed the Frost giants won't hit you if you're moving, but you do take just 1 damage for hitting them. Stopping for just a few frames in a large group can mean a lot of damage.
