Crystals are given to you each level if you're capable in Magic (have 2 Magic or 3 at least). Each three stats in magic you also get on 1 crystal more.
You can also get them from Golem, first time you meet her and I think more times you do meet her, if you give her the Ice Cream and ask for help.
3 soul crystals are given by Warlock once you defeat her, I think. Also 20 crystals are given by Demon King if you have ice cream when you defeat her (but you don't need them already if you can beat her).
Summing it up, you probably get to upgrade once. You either don't have more crystals than that or you need them for magic more than for upgrades.