It's not the bridge. On the world map, go to the south east to the check point at the watch tower. They will let you through after you have permission from the mayor, which you should already have after beating the orc in the cave.
