You're on the right track. But there's still room for optimization yet.
Here's a hint: your trick actually works in every direction, not just backwards.