I think what makes the panning particularly worse is the fact that you're typically panning to the right, where you need to overshoot the fairly thick UI for any movement to happen. The cursor isn't even trapped in the game window either.
My muscle memory from all sorts of software is to hold the middle or right mouse buttons.