I fixed the issue! It was because I had the vJoy driver installed (a driver to use gamecube controllers on pc). Even if I didn't have any controller, the driver was messing up with the game. At least you will know if someone else tells you this problem ^_^
Thanks, seems like it's a unity problem. I checked once the controller using a friend's one and don't have soo much problem but seems like old drivers are creating problems for some reason.