Ahh darn I forgot about that, thanks for the feedback!
With unity you can use GetAxis("Horizontal") and GetAxis("Horizontal") to handle both arrows keys and WASD
Haha yeah looks like that was the problem. I used GetAxis("Horizontal") for the movement originally, but not for the part where stops you from using it. Thanks for the advice!