I can't find the library. I found the school, the monument, and the starting location (which seems like it traps me, and I have to reset the game). Sorry to bug you with issues. I am definitely enjoying the game very much. :)
If I try to go out the way I came from the starting location, it takes me to Green Plaza instead of back to the city map. Then there's a wall on the bottom of Green Plaza.
Speaking with Daphne didn't allow me to enter the castle.
To return to the city map from the Green Plaza, you have to go next behind the temple. Also, pay attention to what Daphne is asking for, that's the next objective.