You have to find a notice in the dungeon and go to the associated lesson. Every lesson unlocks the first spell in a new row
Viewing post in I have a problem, I can't unlock flirt spell, what I have to do?
they look like a piece of paper, the first one is on floor 3, and you need walk over them, then a short scene should trigger about some teachers secret classes. The next time you enter their classroom you can trigger a scene, where you learn a new spell. I don't remeber where the other notices are, but you can find them if you search every floor. Also the next time you enter the library, a scene with a new girl triggers.