Apologies, I fear I misunderstood where you had difficulty, and realized an oversight as well.
If you got stuck in the Crystal cavern, the exit is identified by a funnel of fallen crystals. This requires the ladder from the NPC in the Ruins/Castle area. If you did not speak to the NPC, then you would not have received the ladder, causing you to be trapped, unable to complete the game.
Apologies, again, for the misunderstanding and the oversight if that happened to you.