Only the passive ability "[element] Added " gives you that element on every ability. Otherwise, your element is determined by which ability is active at the time. If you're ever unsure, you can check the status page in the menu, or look next to the character's name in battle to see which elemental affinities they currently have at that particular moment.