When you go out of town there is only one way to get past the wall. Follow the path down the hill and turn left. Once your past the wall you'll need to leave the path and turn right. Follow the outside of the wall all the way and you'll see a cave. Make sure you activate the switch where the closed gate is just before the cave to open a shortcut back to town.