If I try to go out the way I came from the starting location, it takes me to Green Plaza instead of back to the city map. Then there's a wall on the bottom of Green Plaza.
Speaking with Daphne didn't allow me to enter the castle.
To return to the city map from the Green Plaza, you have to go next behind the temple. Also, pay attention to what Daphne is asking for, that's the next objective.