If you're getting blown back, you can't go that way anymore. You need to go find the other three explorers.
From the Quiet Meadow, go to the bottom-left exit. Follow the path left, and then downwards once you reach the cliffs. You should be able to figure out the rest from there.