Was thinking of something similar for mine but got the same problem
Viewing post in A Space Shooter-UDH Jam Week 1 Game jam comments
For a better movement system, I would use a target vector2/3 and a current vector2/3 for movement. The current vector2/3 sets the rb velocity and the current vector is lerped by the target vector with a smoothing of 10ish * deltaTime, thats how I made my movement in scorched polys and it feels really nice