You can, below the selection screen on your journey you see the 8 faces of the gym leaders, on the left of that there is a button and that is used to go back, but that button is only there when you are in the city (at the gym leader selection screen)