Thanks for the valuable feedback.
I agree, since I'm using the default physics, it retain the speed of the previous movement, which makes it slowly moves outwards, I need to try a few things to see if I can fix it while still moving using the physics instead of moving using the transform