Depends on what direction you are headed. If you went East (The Golden Fields' Pass) you must return by travelling West. And vice versa depending on where you are! (South if you went through the Gravel Trail)
I did do that though, and it just kept switching between North and South, even as a test I pressed North once and then South once and I still didn't return and it just said that South was now depart and North was return
it changes between Return/Depart depending on the last direction you went (If you went North, now it becomes Depart and vice versa) So if you want to get back to whitefang you must continue pressing "South" regardless of what it say x3
Also, depending on the text that displays on top you can know how far you are from reaching Whitefang. If you went North once, you must go South twice to get back
