Yeah, I added friction but it did not solved it. Ended up forcing the velocity and angular velocity to 0 whenever it is below a certain minimum velocity and it fixed it :)
