I found a bug where when playing with a controller, you clip into the walls sometimes. (and when it happens controls get reversed for some reason)
Thanks for the feedback, I'll look into it.