For some reason, Unity reads XBox controllers wrong. Because the XBox controllers have analogue triggers, Unity counts them as analogue sticks, and maps walking to them. When I tried with my controller, pressing down the right trigger both moved the camera down and made me move right, while letting go of the right trigger moves the camera up and makes me move left. I'm pretty sure you can fix it in the input tab when launching the game (Although, doing so on my mac causes the program to freeze)