At night enter the haunted forest (through the purple boundary). Head up going through the dangerous area with all the monster girls. After you exit you'll find some cliffs with an entrance further north. Navigate through the obstacles and you'll come upon an abandoned town to the north. Talk to the orc there, she eventually lets you go.