I agree - the haunting storyline is being tweaked a bunch in the next update, which will be out as soon as I can finish it.
And maybe allow you to summon the witch so you can appease the spirits whenever you want.
It'd also be cool to see some benefits for appeasing the spirits, beyond "they won't fuck your shit up" :)