You are right, normal speed loses a bit of sense, but I found the problem, the real reason why it's uncontrollable at normal speed is the impulse mechanic. It's really hard to impulse in the correct direction at full speed.
I'm experimenting with some possible solutions. Thanks for the feedback!