Using time.deltatime shouldn't cause that shake as it's supposed to smooth out movement when frames get lower and when they get higher.
The FPS didn't feel that low so I doubt that was the issue.
