Legendary pieces always have a purple/red name and a colored outline around the piece itself. They often match the effect of a non-legendary piece, but are either stronger or have added effects. You also can't upgrade them at anvils.
I think part of the joy (for me) is changing loot half-way through a run to better fit what's coming at me. You get a lot of the material you spent upgrading back, so it is sometimes advantageous to change!