I don't remember exactly how to get the charm spell. I know you get it from Selena at some point. I think you need to finish the toy maker questline first, (where he makes his doll wife Candy after you feed the cat in the plaza). After that, Selena will ask you to find a locket or something in the toy shop. Talk to the doll to get it, then Selena will give you the charm spell. (I might be missing a step here or there, it's been a while.)
To first encounter the demoness, you need to explore the dark woods with Evie in your party. Keep exploring and you'll eventually get a cut scene where you find some ruins.
After that, you can encounter the demoness again by exploring the dark woods (you can do this alone now if you want) until the forsaken keep area opens up.