I designed the movement to be that way to be analog to a real ship's movement. To make it more controlled I implemented the X button, which slows you down, at a cost: you are easier to hit.
Maybe a way to fix this problem would be to add a fixed top-speed, in which when it is reached, no more accelerating is done.