Hi! Yes, you can add or subtract from your SONG/SWORD number. I refer to it as your "number" for short, since its the only one you roll against.
It's more sliding along a continuum than getting better or worse. Move up and you get better at SONGS; down and you get better at SWORDS.